Augmented Vision-Language Models: A Systematic Review
Authors: Anthony C Davis
Burhan A. Sadiq
Tianmin Shu
Chien-Ming Huang
Venue: Accepted by TMLR ()

Abstract: Recent advances in visual-language machine learning models have demonstrated exceptional ability to use natural language and understand visual scenes by training on large, unstructured datasets. However, this training paradigm cannot produce interpretable explanations for its outputs, requires retraining to integrate new information, is highly resource-intensive, and struggles with certain forms of logical reasoning. One promising solution involves integrating neural networks with external symbolic information systems, forming neural symbolic systems that can enhance reasoning and memory abilities. These neural symbolic systems provide more interpretable explanations to their outputs and the capacity to assimilate new information without extensive retraining. Utilizing powerful pre-trained Vision-Language Models (VLMs) as the core neural component, augmented by external systems, offers a pragmatic approach to realizing the benefits of neural-symbolic integration. This systematic literature review aims to categorize techniques through which visual-language understanding can be improved by interacting with external symbolic information systems.

Published in Transactions on Machine Learning Research (01/2026)
Augmented Vision-Language Models: A Systematic Review
Anthony C Davis
Lony, danis Dewaaaaaisaaalll.oo edu
Johns Hopkins University
Burhan Sadiq Johns Hopkins University
bsodiq1 @jhu edu
Tianmin Shu
Shanumnin..suuueeee edu
Johns Hopkins University
Chien-Ming Huang
Creemmming.huann@@iiiii edu
Johns Hopkins University
Reviewed on OpenReview: https: optp://veeerr.eee...mm net/ forum? id=DFMP177u6.
Abstract
Recent advances in visual-language machine learning models have demonstrated exceptional ability to use natural language and understand visual scenes by training on large, unstruc- tured datasets. However, this training paradigm cannot produce interpretable explanations for its outputs, requires retraining to integrate new information, is highly resource-intessive, and struggles with certain forms of logical reasoning. One promising solution involves inte- grating neural networks with external symbolic information systems, forming neural sym- bolic systems that enhance reasoning and memory abilities. These neural symbolic systems provide more interpretable explanations to their outputs and the capacity to assim- ilate new information without extensive retraining. Utilizing powerful pre-trained Vision- Language Models (VLMs) as the core neural component, augmented by external systems, offers a pragmatic approach to realizing the benefits of neural-symbolic integration. This systematic literature review aims to categorize techniques through which visual-language understanding can be improved by interacting with external symbolic information systems.
1 Introduction
1.1 Motivation
Vision-Language Models (VLMs) represent significant leap forward in artificial intelligence (AI), showing remarkable abilities to interpret complex visual scenes and generate coherent natural language descriptions, powering advancements in tasks such as visual question answering (VQA) (Alayrac et al.) 2022 and im- age/video captioning (Radford et al. 2021). Trained on vast web-scale datasets, these models excel at mapping between visual inputs and textual concepts. However, this end-to-end training paradigm inherently limits their capabilities several critical ways. VLMs produce outputs without clear justifications, making them difficult trust or debug without specialized tools (Rudin et al.) 2021 Stan al.) 2024). Integrating new factual knowledge or correcting errors typically requires resource- resource-intensive intensive retraining. Furthermore, despite their semantic understanding, VLMs often struggle with tasks that require precise logical deduction, mathematical calculation (for example, accurate object counting), verifiable factual recall of entities within an image, and complex spatial reasoning (Khajuria et al. 2024 Zhang et al.) 2025b). These limitations hinder their deployment in high-stakes applications that require precision, reliability, and adaptability. The concopt of augmenting VLMs with external information systems has evolved into several distinct paradigms that offer practical solutions to VLM limitations. These augmentation approaches can be broadly categorized by the type of external resource utilized and how it interfaces with the VLM.

---

Published in Transactions on Machine Learning Research (01/2026)
Retrieval-based augmentation has emerged of the most widespread approaches, with Retrieval Aug- mented Generation (RAG) (Lewis et al. 2021 becoming common paradigm in both research and commer- cial applications. These methods retrieve relevant information from external sources to provide context for the VLM's processing. The retricval mechanisms vary considcrably: dense vector-basee retricval uses learned embeddings to find semantically simílar content, oflen employing pretrained encoders like CLIPP or custom embedding models; traditional term-based methods like BM25 provide lexical matching capabilities; and structured retrieval from knowledge graphs enables access explicitly encoded relationships and facts (See Figure 1 for example of knowledge graph retrieval). While some approaches fine-tune the VLM jointly with the retriever to improve retrieval relevance and integration (Chen et al. 2022b Rao et al. 2023 Yuan et al. 2023b), many implementations use frozen VLMs with off-the-shelf retrieval systems, demonstrating the flexibility of this augmentation stralegy. The retrieved information can be integrated at different slages: as additional input context (prompt augmentation), during the model's reasoning process, or to validate and refine outputs.
Feature extraction module
Base attention modules
Objects
Apple
Edge Medale
Table
Adaptive Score Atteniion Module
Stem
Image
Clasifir
Why there lisifo on the apple?
GloVe LSTM
Canal Allention Medile
Question
Apple is related to fruit.
ConcrpNec
An apple can be caten.
A kmiit out that apple.
GLoVe LSTM
ldge Attanion Modale
cutting
Desk related to tible.
Apple is reiuted
Stem
Knowledge
External knowledge
Figure 1: Architecture of the Knowledge-based Augmentation Network (KAN) by Zhang et al. (2020) The system extracts visual features via an object detector module and retrieves external knowledge from ConceptNet with labeled relationships and reliability scores.
Symbolic computation augmentation represents another major category where VLMs interface with external computational tools and reasoning engines. Program synthesis approaches enable VLMs to generate exe- cutable code (e.g., Python scripts, SQL queries, or domain-specific languages. See figure 2 for an example) that operates on structured representations or queries external systems, with the execution results informing the VLM's outputs. Symbolic reasoning engines such logic solvers, planning systems, and specialized rea- soning frameworks can be invoked to perform precise logical operations that complement the VLM's pattern recognition capabilities. The rapidly evolving paradigm of tool use (Schick et al.) 2023 Qin et al. 2023) treats diverse external capabilities (calculators, APls, specialized vision modules, web browsers) as tools that the VLM can dynamically invoke based on task requirements. Additionally, symbolic graph operations allow VLMs to manipulate structured representations like scene graphs or knowledge graphs through operations such as graph traversal, node matching, or relational rcasoning, bridging percoptual understanding with structured symbolic manipulation.
These augmentation strategies offer pragmatic path forward by building upon the sophisticated visual and language understanding already present in statc-of-the-art VLMs, rather than replacing them. Through augmentation, a single VLM can be adapted to diverse tasks without needing to master every capability internally. Importantly, VLMs can be trained when and how to invoke these external resources, discovering effective strategics for combining their internal representations with external capabilities. This approach enables targeted mitigation of specific weaknesses, such as mathematical computation through calculators, factual accuracy through knowledge bases, and complex spatial reasoning through specialized geometric
2

---

Published in Transactions on Machine Learning Research (01/2026)
Query q
Visual Input æ
"Which pet is in the top left?"
ViperGPT
Generated Code 2
function(inage) inage_patch = InagePatch (Inag)
Code LLM π
image_pated find
API Specification
Code Execution &
(mpsi(immge) teach Serser:
Python Interpreter
def estata(inage digect, nume) boik:
(leeyDhaa) sieclartty lmagere
API Implementation
Result:
"Shiba Inu"
Figure 2: The ViperGPT framework by Surís el al. (0023) Given a visual query and an image/video input, ViperGPT uses a code-generation model (GPT-3 Codex) generate Python code that composes various vision modules through API. The generated program makes explicit function calls to vision capabilities (e.g,, find, pempute_deptaan count) and uses Python's built-in logical and mathematical operators to reason about the results.
modules (Surís et al. 2023 Marino al. 2020 Yi al. 2018). This systematic review examines how these augmentation techniques realized in practice, analyzing their implementation strategies, capabilities, and the specific VLM limitations they address.
1.2 Augmented Vision-Language Models: Definition and Scope
We define an Augmented Model as a system where external information or computational processes are actively integrated with neural model's inference operations (before, during, or after its forward pass) enhance its capabilities. This survey focuses specifically on inference-time augmentation, where external resources are accessed during model execution, not training-time data augmentation. However, the AVLMs survey may still undergo finetuning to improve their ability to utilize these external resources effectively. This inference augmentation is distinct from prompting techniques like chain-of-thought or in-context ex- amples (Wei et al.) 2023; Zhao et al. 2023a which elicit latent reasoning capabilities without accessing external data or tools.
A Vision-Language Model, for the purpose of this review, a machine lcarning model that jointly processes and understands visual and textual modalities, either through generation tasks (e-g., VQA, captioning), alignment tasks (e.g., image-text retrieval, zero-shot classification), or both. This encompasses models that output natural language text as well as those that learn joint representations of vision and language.
Therefore, an AVLM is an VLM integrated with external symbolic information systems, APIs, databases, or other computational tools. An AVLM may involve modifications in VLM neural architecture, or it may involve pre- post-processing of inputs or outputs of the VLM. Regardless, these integrations aims over- come the inherent limitations of standalone VLMs and represent particularly compelling implementation of the augmented neural system concept. A glossary of key terms in this survey is included in Appendix C
3

---

Published in Transactions on Machine Learning Research (01/2026)
1.3 Related Work and Knowledge Gap
The quest to enhance neural models, particularly in the vision-language domain, by incorporaling external knowledge or symbolic reasoning has spurred significant research, reflected in several existing surveys. Re- views on knowlodgg-onhancke multimodal learning (Lymperaion & Stamou 2022: Zhao et al., 2023b Wajid et al. 2023) investigate integrating factual knowledge, often via knowledge graphs or retrieval augmenta- tion, to improve tasks like captioning and VQA. Concurrently, surveys exploring neuro-symbolic approaches [Aditya et al., 2019; Senior et al., 2023 Hitzler et al.) 2022 Khan et al. 2024] examine the broader chal- lenge of combining neural perception with symbolic reasoning, often focusing on graph neural networks, spalio-tempp logic, or commonsense knowledge integration for better scene understanding and reasoning. Specific arcas like VQA have also been surveyed (Jamshed & Fraz] 2021 Mostafa et al., 2020), tracing the evolution towards models capable of more complex reasoning, sometimes touching upon the need for external knowledge or structured representations.
While these surveys provide valuable context by covering knowlodge integration, ncuro-symbolic mothods, and advances in VQA reasoning, they do not specifically offer systematie review focused on the augmenta- tion of VLMs through interaction with diverse external symbolic systems and tools. Existing reviews often focus on specific knowledge types (e.g., knowledge graphs) or broader neuro-symbolic theory. There is a knowledge gap in understanding the landscape of techniques specifically designed to connect modern VLMs with external symbolic resources in a flexible, often learned manner (i.e., tool use). Particularly, there is a lack of systematic analysis regarding how these augmentation techniques address core VLM challenges, such as their noted difficultics with procise spatial reasoning (Wang et al. [024c] Zhang et al. 2025b). Aug- mentation via external tools or information sources presents polential pathway to compensate for such weaknesses by providing structured spatial information or enabling interactions with geometric reasoners, least until VLM architectures intrinsically improve in these areas.
This systematic literature review aims to fill this gap by specifically categorizing and analyzing techniques where VLMs interact with external symbolic information systems or tools to enhance their vision-language understanding capabilities. We seek to provide a structured overview of how these augmentations are imple- mented, what types of external systems are used, and how they address the limitations of standard VLMs, with a particular interest in emerging tool-use paradigms and their application to challenging visual reasoning tasks.
2 Overview: Three Stages of Vision- Fusion
The papers surveyed demonstrate a variety of techniques for augmenting vision-language models with ex- ternal symbolic information systems. The selection of these studies is the result. of a systematic literature search conducted according to the PRISMA guidelines (Page et al. 2021 J, which involved querying academic databases with specific keywords and applying rigorous inclusion /etusiio/eexussiioo exclusion criteria to identify relevant pub- lications (see Appendix A This process ensures that the surveyed works specifically target inference-time augmentation and filter out approaches like pure prompting or training-time knowledge integration. To structure this diverse landscape, we categorize the surveyed approaches based on three key characteristics:
When the external interaction occurs relative to the VLM's processing pipeline. We distinguish be- tween Early Fusion (integrating external data at the input stage, influencing initial represcntations), Middle Fusion (interfacing with external systems during the VLM's internal reasoning or generation steps), and Late Fusion (using the VLM's initial output to trigger external processing, validation, or refinement).
What type of external information or computation is leveraged. This includes Retrieval (accessing pre-existing facts or knowledge from sources like knowledge graphs or text corpora) and Symbolic Computation (generating new information through logical deduction, program exccution, or special- ized computational tools), or a combination of both.
How the fusion is specifically implemented, detailing the particular mechanisms used in each ap- proach.
4

---

Published in Transactions on Machine Learning Research (01/2026)
This review primarily organizes findings according to the temporal fusion stage (When), this significantly impacts how external information influences the VLM. Within each temporal category (Farly, Middle, Late), we further analyze the type of external interaction (What) and discuss notable implementation details (How). While some sophisticated methods may blend characteristics, this framework provides a structured lens for comparing the underlying principles, capabilities, and trade-offs of different augmentation techniques. The following sections claborate on the findings for each category, referencing the detailed categorizations presented in the Appendix tables (Tables 2 through 5
3 Early Fusion Methods
Early fusion methods augment the VLM by incorporating external information directly at the input stage, before the core VI.M begins its internal processing. This is often the conceptually simplest approach, treating external information as additional context and potentially requiring advantage is implementation simplicity, offering a direct way to provide context. However, it faces challenges related to the relevance and noise of retrieved information. For example, some implementations use generated image captions as retrieved context which may introducc information loss. The choice between simple prompt augmentation and more structured retrieval encoding depends on the desired level of integration and complexity tolerance. These methods primarily fall into retrieval-basee or, less commonly, symbolic computation-based categories, as detailed in Appendix Table
VLM architecture changes. Its main
3.1 Retrieval- Based Early Fusion
The most common carly fusion strategy involves retricving relevant information from external sources and providing it alongside the primary visual and textual inputs. A primary technique is Prompt Augmen- tation, where retrieved textual context is directly appended to the input prompt, exemplified by Retrieval Augmented Generation (RAG) (Lewis et al. 2021). This retrieved text can originate from various sources. Text/Fact Retrieval draws information from text corpora or knowledge graphs (KGs), using approaches ranging from pre-trained encoders like CLIP without further training to fine-tuning the retriever, possibly jointly with the VLM, for better relevance (see Table column "Retrieval FT*). See Figure 3 for an example of text retrieval using a pretrained vision-language encoder. Reranking retricved results often employod to enhance quality (Qu et al.) 2024 Liu et al., 2024 Wen et al.) 2024). Retrieved KG triplets can also be formatted as text for the prompt (see Table 2 column 'Prompt. Augmentation"). An alternative form of prompt augmentation uses Image Caption Augmentation, where textual descriptions (captions, labels, Optical Character Recognition (OCR)) are first generated from the visual input, and this text is then used for retrieval or directly added to the prompt, with some methods jointly training the caption generator and retriever (sce Table 2 column "Image Caption"). While simplifying the problem to text-based retrioval, this approach risks information loss during captioning.
Instead of appending raw text, another approach uses Retrieval Encoders to encode the retrieved informa- tion (c.g., KG subgraphs, text passages) into separate embedding vectors. These embeddings then condition the VLM, often through attention mechanisms (Yuan et al.] 2023b Weng et al. 2024 Chen et. al. 2022a Salemi et al. 2023a). Long Short Term Memory models (LSTMs) (Wu et al.) 2016 or memory modules (Hu et al. 2022 This allows for a more structured integration of knowledge. Specifically, KG subgraphs can be encoded using Graph Neural Networks (GNNs) (see Table 2 column 'Subgraph Enc") or fused with scene
graphs (see Table 2 column "KG Conv"). Multimodal KGs can also provide richer representations (Jiang &
Meng 2023
3.2 Symbolic Computation Early Fusion
Integrating the results of symbolic computations at the input stage is rare in the surveyed literature. The primary example identified (Potapov et al.] (2019) involves transforming the visual input into symbolic scene graph. This structured representation, potentially processed by an external symbolic reasoning engine like OpenCog, serves input conditioning for the VLM. This approach explicitly introduces symbolic structure early on but depends heavily on robust perception-to-symmool conversion modules.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Clanec Leat ailoees.
CXR-ReDonE (ALEF contrastively pre-trained Test Encoder)
corsesstion...
FAISS Vector Database
Full CXR-PRO
Text Vector Embeddings
Report Corpus
INDEXING
(Impression sedioons only)
UT
siillartty
product
Prompt Builder
FAISS Retriever
OpenAI LLM
Instructions:
Top impressions
teat-dawinc--03
contest
got-3.5 turbo [ChatGPT]
report
Context:
Image Vector Embeddings
Impressions
OpenAl
generated Impression
CXR-ReDonE (ALBEF contrastively pre-traized Image Encoder)
Impressions
Response Synthesizer
[Optional for larger context]
INFERENCING
sporl pows geare pruna etusor'
output Impression
Query Image
Figure 3: Architecture for Retrieval Augmented Chest X-Ray Report Generation by Ranjit al. (0023) Text embeddings from radiology impressions indexed in vector database. For an input X-ray image, its embedding, generated by contrastively pretrained vision-language encoder (CXR-ReDonE), is used to retrieve the most similar text (impressions or sentences) from the database. This retrieved text then forms the context for a prompt, along with specific instructions, which is fed to an LLM (e.g., OpenAl GPT models) to generate the final radiology report impression. This process illustrated for both indexing and inferencing stages.
4 Middle Fusion Methods
Middle fusion techniques integrate external information or symbolic computation during the VLM's infer- ence stage, allowing interaction with the model's intermediate representations before the final response is generated. Intermediate representations embedding- based in dense retrieval techniques. This enables more dynamic and potentially iterative integration compared to early fusion, where external data influences internal processing, reasoning steps, or feature refinement. By allowing external information and symbolic processes to interact with the VLM's internal state, these methods enable context-aware reasoning and iterative refinement. This often involves more complex architectures and training but holds promise for leveraging both neural pattern recognition and symbolic manipulation more effectively. The rise of tool use and agent-based frameworks within this category points towards VLMs acting less as monolithic predictors and more as components in larger rea- soning systems, echoing paradigms like Kahneman's System (neural intuition) and System 2 (deliberate symbolic reasoning) [Kahneman 2011 Booch et al. 2020). These methods, categorized in Appendix Table g often involve foedback loops or specialized modules operating alongside main VLM components.
mean either token-based such as in autoregressive generation,
4.1 Retrieval-Based Middle Fusion
These methods retrieve external information based on intermediate VLM states and fuse it back into the ongoing computation. One approach is Dense Retrieval, which uses dense vector similarity between in- termediate VLM representations and a knowledge corpus to find relevant information (often images or text) that is then fused back into the model's layers, typically via attention (Wang et al.j 2022b Lin al. 2023b Jia al. 2023). Another major approach leverages Graph-Based Retrieval, primarily using KGs. This includes methods where intermediate visual or textual features trigger KG Querying: the retrieved

---

Published in Transactions on Machine Learning Research (01/2026)
subgraphs facts are processed (often with GNNs) and fused with VLM representations, sometimes after extracting visual subgraphs first (see Table 3 column 'KG Prompt Augmentation".. Figure illustrates a middle fusion approach that constructs coupled scene and concept graphs, using shared entities as mediums for cross-modal knowledge exchange. Other graph-based mothods use Similarity Mcasures between inter- nal VLM representation and KG elements to guide reasoning or weighting, rather than directly injecting
KG structure (Wu al.] 2024a; Chac Kim 2022 Li et al. 2019 ming Xian et al.
2023 Marino et al.
2020). A significant group focuses on Concept/ Scene Graph Fusion, explicitly combining internally gen- erated scene graphs with external concept graphs (e.g., from ConceptNet (Speer et al.
), often using GNNs on the combined graphs (see. Table column 'Concept/Scene Fusion "). More complex structures like
2018
Multimodal KGs (MMKGs) (Xi al. 2024 Shi et al. 2022 Santiesteban et al. 2024 Ouyang et al. 2024
Liu et. al., 2021) or Hypergraphs (Heo el al., 2022 Wang el al. 2024b) are also integrated using specialized
graph networks. Finally, Reinforcement Learning (RL) can be used to learn policies for querying or
integrating external knowlodge basod on the current state (Bougic ct al. 2018
Mhat type of temperature is this?
Controt Rrceding
Thel
Concept Graph
Probubility Score
Pecudo-Stamese Graph Medium Furicn
Dasoal
CIRE
Scene Grap
Twn-Stage Prompting Strategs
The MAIL (Modality- -wwae Aware Integration with LLMs) framework by Dong et al. (024 The system employs two-stage prompting strategy: first generating a dense caption through visual LLM, then constructing a scene graph by extracting spatial and object features as triples (e.g., (person, wearing, coat)). These scene graph entities are linked with external knowledge from ConceptNet to form a coupled concept graph containing real-world facts (e.g.: (coat, used for, warn)). A pseudo-siamese graph medium fusion module processes both graphs through parallel graph attention networks with different weights, using shared mentioned entities as mediums enable cross- moosl exchange while preserving intra-modal information.
Figure 4:
4.2 Symbolic Computation Middle Fusion
These methods incorporate symbolic reasoning, calculations, or tool use within the VLM's processing pipeline. One key technique is Program Synthesis, where the VLM generates intermodiate programs (e.g., functional programs, Python code) operating ou symbolic input representations or querying external
tools; the execution result influences subsequent. VL.M processing (Zhang et al.] 2022c 2023e
Hu et al.
2023b), (Shirai et al.] 2023]. see Figure 5), (Zhang al. 2023b Li et al.||2211] Mishra et al. 2024 Xue al,
2024h. Another approach involves integrating Symbolic Logic Engines, translating intermediate VLM representations into facts or queries processed by engines like differentiable first-order logic (Zhang et al.
2025a), Answer Set Programming (ASP) (Riley & Sridharan 2019 Mitchener et al.
2021 Description Logic 2022b 2023d), temporal
Tsatsou et al.] 2021). planning domain definition languages (PDDL) (Zhang et al.
logic CChii et al. 2024), specialized neurosymbolic languages like Scallop (Li et al. 2023d Huang et al,
2021 1, or embedding propositional logic operations (Li et al.] 2023c). Vector Symbolic Architectures
(VSAs) represent symbols and perform operations using high-dimensional vectors within the neural archi-
tecture Montone al., 2017 Kovalev et al., 2021). Some methods perform Symbolic Graph Operations
directly graph representations (scene graphs, KGs) during processing, like guided walks routing (Li
ct al. 2022c Liang et al.) 2020 Wu ct al. 2023 Zhao 2015 Yang et al.) 2020 Zhang et al. 20231 Hudson
& Manning 2019 Cao et al. 2021). Increasingly popular is Tool Use, where the VLM dynamically calls external tools (calculators, APIs, vision algorithms, drawing tools) based on its intermediate state, integrat-
ing the tool's output (see Table 3 column "Tool Usc"). Lastly, Sclf Play involves using the VLM within a simulated environment where it interacts, uses tools (potentially itself), and learns from feedback (Misiunas)
et al. 2024
7

---

Published in Transactions on Machine Learning Research (01/2026)
Scene Observation
Linguistic Instruction
Input
"Slice the carrot and place in the bowl." "
Vision-Language Interpreter (ViLain)
Proposed Framework
V&L Models
Textual Scene Description
Domain Knowledge
Object characteristics
Input/output examples
LLM
Re-Prompting
<<problem.pdoe
Err. Msg.
Symbolic Planner
Output
1. objects
or
2. initial state
Valid
3. goal
Plan
Problem Description
Figure 5: Overview of the ViLaIn approach for VLM planning of robotic actions Shirai et al. (002) The vision-language interpreter (ViLaIn) generates a problem description from a linguistic instruction and scene observation. The symbolic planner finds an optimal plan from the generated problem description.
4.3 Combined Retrieval and Symbolic Computation Middle Fusion
These advanced methods integrate both retrieval and symbolic computation during the forward pass. Many employ Agent architectures where the VLM acts controller, deciding when retrieve information and when to symbolic tools (including sub-agents code execution) to achieve goal (see Table 3 column 'Agents'). Other Approaches combine retrieval (e.g., from ontologies, KGs) with symbolic reasoning (c.g.. probabilistic logie, program synthesis, graph walks, concept binding) in bespoke ways for specific tasks like
embodicd QA, riddle solving, or rumor dctection [Besbes et al. 2015 Aditya et al. 2016 Aditya
2017
Aditya Baral 2016 Tan el. al. 2021 Liu et al. 2023a Stammer et al. 2024 Valashsky & Ullman 2018
Gao et al. 2023c 2024).
5 Late Fusion Methods
Late fusion methods apply external information retrieval or symbolic computation after the VLM has gen- erated initial output. This external step typically serves to validate, refine, explain, augment the VLM's output using structured knowledge precise tools. Late fusion provides powerful mechanism for verification, refinement, and explanation by applying structured knowledge or precise computations to the VLM's generated output. It leverages the VLM's ability to produce a plausible initial response, which then guides a more targeted external process. This approach is particularly well-suited for enhancing reliability and interpretability, as symbolic steps can act as explicit checks or provide traceable reasoning paths. The main dependency is the quality of the initial VLM output; if it too vague or incorrect, the subsequent external process may be misguided. These techniques cataloged in Appendix Table
5.1 Retrieval- Setree Late Fusion
Here, the VLM's output triggers a targeted retrieval query. In Dense Retrieval, the initial VLM output (e.g., answer, rationale) queries dense retrieval system. The retrieved information (text, facts) then used
8

---

Published in Transactions on Machine Learning Research (01/2026)
to refine the output provide supporting evidence (Song et al. 2022a b Shi et al. 2024 Alternatively,
using Knowledge Graph Retrieval, the VLM's output (e-g., generated caption, predicted relationships) queries a KG. Retrieved facts or subgraphs refine the output, for instance, by adjusting probabilities or improving relationship predictions (Goo et al. 2022b
Huang et al. 2020 Xiao & Fu 2022
5.2 Symbolic Computation Late Fusion
This involves applying symbolic tools logic engines to the VLM's output. Program Synthesis generates programs based on the VLM's output for analysis, validation, transformation. Examples include gener- aling Python code to verify VQA answers via vision APIs, treating symbolic programs as latent variables, or generating Structured Query Language (SQL) queries from the output (see Table 4 column 'Program Synth"). The influential Ncural-Symbolic VQA (NS-VQA) approach (Yi ct al. 2018) executing programs on scene representations post-pprdiciion, often adapted. Symbolic Engines feed the VLM's output (or derived symbolic representations) into formal logic engines (e.g., Prolog, ASP, Probabilistic Soft. Logic) for
consistency checking, inference, or validation (Sethuraman al. 2021 Aditya et al.) 2018 Eiter et al.) 2022; 2021 Cunnington et al. 2024), use PDDL for planning (Xu et al. 2022 Tool Use involves calling exter-
nal lools or APIs based on the VLM's output for specialized lunctions, verification, generating structured
data Yuan et al. perform manipulations on graph representations derived from the VLM's output, such as reasoning over action chains or graph traversals (Li et al.
2023a Cesista et al. 2024 Cesista
2024 Zhang 2023). Symbolic Graph Operations
2023a Zhan et al. 2021 Saqur & Narasimhan 2020 Johnston
et al. 2023). Other Approaches include applying symbolic solvers to latent representations (Singh 2018
using VLM output confidence to trigger human interaction or further symbolic checks (Bao et al. 2023
updating conversational memory based the response (Verheyen al. 2023
5.3 Combined Retrieval and Symbolic Computation Late Fusion
These methods combine both retrieval and symbolic computation after the initial VLM output. Typically,
the VLM output is parsed into a logical form, relevant domain knowledge (facts or programs) is retrieved, and a symbolic reasoner (e.g., probabilistic logic, ASP) derives the final answer (Sachan 2020 Basu el al.
2020). The AQuA framework (Basu et al., 2020) depicted in Figure
Preprocessor
YOLO
SRE
Query Generator
Common Sense Knowledge
Knowledge
(FACTS & RULES)
Semantic
(FACTS)
Relations
s(ASP) Engine
Query
Answer
Figure 6: The architecture of the AQuA framework Basu et al. 2020 It consists of five main modules: (i) YOLO for object detection and feature extraction, (ii) Preprocessor for the natural language question, (iii) a Semantic Relation Extractor (SRE), (iv) Query Generator based on semantic analysis, and (v) a retrieval based Commonsense Knowledge module leveraging. The system utilizes ASP engine for symbolic reasoning.
9

---

Published in Transactions on Machine Learning Research (01/2026)
6 Discussion
The studies reviewed in this paper underscore the growing success of incorporating external symbolic in- formation into vision-language models across different fusion paradigms. Here we discuss key observations, challenges, and potential directions for future research stemming from these findings.
6.1 Domain-Specific Limitations and Augmentation Solutions
The integration of external symbolic systems in vision-language models addresses fundamental perceptual and reasoning limitations that distinguish visual understanding from purely lextual tasks. We organize domain-centrie limitations of VLMs along five phenomena that recur across the surveyed studies: spatial, temporal, knowledge grounding, physical commonsense, and action netion/cmbert /embodiment (STKPA). Each category of limitation has driven the development of specific angmentation patterns that leverage the complementary strengths of neural perception and symbolic computation. Orthogonal to this axis is how external structure is injected, via retrieval, program /tool tool execution, or logic/constraint checking, and when it is injected (early, The matrix in Table summarizes typical augmentation patterns observed for each phenomenon. Across the 264 papers our corpus, each work is aligned with at least. one STKPA category, as shown Tables to 4 We will discuss each domain and highlight some example augmentation solutions and their associated datasets.
middle, or late fusion; see Sections 3 5)
6.1.1 Spatial numeracy and geometry
VLMs struggle with tasks requiring exact measurements or counts within visual scenes, with error rates exceeding 30% simple counting tasks (Wang el al. 2024c). This limitation stems from the continuous nature of visual features in neural representations conflicting with discrete spatial reasoning requirements.
Effective augmentation systems externalize discrete structure through two primary familics of approaches. First, program synthesis methods compile questions to programs that operate on structured scene repre- sentations. Neural-Symbolic VQA (Yi et al.) 2018)) achieved 99.8% accuracy on CLEVR through explicit program execution over scene graph representations, effectively saturating the benchmark with 15+ per- centage point improvement over end-to-end neural baselines. Second, visual API orchestration methods like ViperGPT (Surís et al. 2023) generate Python code that composes heterogencous computer vision modules such as detectors, bojjct/semannt bolic information about the visual scene. This compositional approach allows VLMs to leverage specialized perception modules' strengths while maintaining flexibility through learned orchestration policies.
segmenters, or depth estimators, each providing complementary sym-
Spatial reasoning benchmarks have been fundamental in evaluating AVLMs' ability understand geometric relationships and counting (Table 5) The CLEVR family of datasets (Johnson et al.) 2016) provides syn- thetic but precisely controlled evaluation of spatial reasoning, with extensions like Haper-CLEVRR (Li et al. 2022cb testing domain robustness and CLEVR-POC (Abraham et al. 2024) introducing partial observa bil- ity challenges. Scene graph datasets including Visual Genome (Krishna el al. 2016)) and VCD (Shen el al. 2024) evaluate models on real-world spatial relationships between objects. More recent benchmarks like SOK-Bench (Wang et al.] 2024a) combine spatial and knowledge requirements, while specialized datasets test whether VLMs can reason about times and locations (Zhang et al. 2023a or answer complex visual information-seeking questions (Chen al.] 2023
6.1.2 Temporal and causal ordering
In video understanding, VLMs frequently misinterpret temporal sequences and causal relationships, with accuracy drops of 20-40% on tasks requiring temporal ordering compared to static image reasoning (Yi et al. 2019). The absence of explicit temporal reasoning mechanisms in standard architectures necessitates augmentation with structured temporal representations.
Middlo-fusion pipelines address this by tracking entities and actions through specialized tools (detection, tracking, pose estimation, optical flow) construct event sequences that programs or planners can reason over. Logic-based approaches apply temporal operators and constraint checks to predicted timelines, with
10

---

Published in Transactions on Machine Learning Research (01/2026)
Retrieval
Symbolic Computation
Spatial
KG fact retrieval to bias relations/attribuu Executable visual programs over scene graphs (NS- al.
(Marino al. 2020 Li et al. 2020 ; dense
VQA) (Yi
2018); Python tool orchestration
gion/sto Seenteene retrieval for spatial cues (Chen
for counting/ geometry
/depth /dephh)
2022b Lin
2022); graph-based
(Suris et 2023 visual program distillation (Hu
trieva of related objects/relations relations (Wang et al.
et 2023b
ASP
/logic verification of spatial pred-
2022c
Zhu el 2020b
icales (Basu et al.
2020
Eiter al. 2022 Sethu-
raman el al. 2021
probabilistic neural symbolic
constraints (Vedantam et al. 2019
Temporal
Knowledge-guided caption/entin retrieval for event context (Xi et al. 2024 Hou al. 2020);
LLM-orchestrated video
tools (tracking,
OCR/ASR, shot segmentation) (Fan al 2024
ontology lookup of task fask/event relations
[Jiang
program traces over events (Zhang et al. 2023e]
et al. 2020
symbolic activity reasoning with tool-angmented
execution (Wu et al. 2023); temporal logic/order
constraints and ASP checks (Choi
2024
Eiler et al. 2022
Knowledge Outside-knowledgs retrieval (text/KG, rerank- grounding
Code/program generation to query KGs/APt compose operators (Subramanian et al. 2023
ing) for VQA (Marino et al. 2019
Chen
al. 2022b
Wen et al. 2024]; multi-source
Zhang 2023 agentic tool-use for information seek-
multimodal
retrieval (entitics, captions, KG)
ing (Hu et al,
2023c]: PSLL/formal-loo consistency
(Zhu al.
2020b
Salemi al. 2023a Gui
and entailment (Aditya al. 2016 Sethuraman
et al.
2021
et al.
2021
Eiter et al. 2022
Physical common-
Affordance/ ffunctiona KB retrieval (e.g., Con- Differentiable/explichi physics for feasibility (Huang al.) 2023); vision-tool composition for metric/- geomelry checks (Surís et al. 2023); 3D symbolic grounding of objects/relations (Hau et al. 2023
ceptNet) (Speer al. [018 Marino et al.
2020
sense
knowledge-aided captioning Huang et al.
2020
rule/constraig checking for stability, containment, and causal function [Aditya et al. 2018
Vedantam
et al.
2019 Zhao 2015)
Action embodi- ment
External maps/kmon haaps/knowledge for navigation and planning (Li et al. 2022d Ni et al.) 2023); VL-
Planner interfaces (PDDL. ASP)
invoked from
VLM outputs (Shirai et al. 2023 Zhang et al. 2022b);
action models transferring web knowledge for
plan-execute-observe loops grounded by VLMs Zhang et al. 2023b; Gao et al.
control (Brohan et al. 2023;; robotic manipu-
2023b
lation knowledge bases
Gao el 2023b
documentation for zero-shol tool use (Hsieh et al.
2023 planner validity/safetyy constraints and neuro-symbolic agent frameworks [Xu et al. 2022
Cunnington al. 2024
Table 1: Reasoning domains and associated augmentation mechanisms with example implementations
temporal logic systems (Choi et al. 2024 showing 15-25% improvements on temporal ordering tasks. Re- trieval of script knowledge or prototypicn event progressions biases hypotheses toward plansible sequences, while lato-fusion verifiers test temporal fcasibility against these priors. These augmented approaches sepa- rate perceptual feature extraction from logical temporal inference, allowing each component to operate in its optimal representational space.
(Table 5 which extends Temporal reasoning evaluation is primarily anchored by CLEVRER (Yi ct al. 2019
spatial reasoning into the temporal domain through collision events and causal chains in synthetic videos. The Compositional 4D dataset (Wang et al. 2024d) further challenges models with nour-dimensional scene understanding requiring both temporal and physical reasoning. While fewer dedicated temporal datascts exist compared to other domains, temporal reasoning is often implicitly tested embodied agent benchmarks and video understanding tasks that require tracking state changes over time.
11

---

Published in Transactions on Machine Learning Research (01/2026)
6.1.3 bhowledge-intensive grounding
The challenge of linking language to specific visual entities requiring external knowlodge (identifying land- marks, recognizing famous individuals, understanding fine-grained categories) represents a fundamental lim- itation where learned pattern matching proves insufficient due to sparse training data for highly specific instances (Kalai et al. )025).
Multimodal knowledge graph navigation addresses this through explicit retrieval and graph-based reasoning. VQA-GNN (Wang et al. 2022c) demonstrated 4.6% improvement on GQA by coupling internally generated scene graphs with external concept graphs through graph neural networks that enable message passing between visual features and symbolic knowledge. KAT (Gui et al. 2021) achieved 53.1% on OK-VQA, a 5.1% improvement over PICa which relied solely on GPT-3's parametric knowledge (Yang et al. 2022 The integration of structured knowlodge graphs like ConceptNet enables more accurate entity recognition through explicit retrieval of facts about identity, function, and long-tail calegories, rather than relying solely learned pattern matching.
Knowledge-based VQA represents the largest category of benchmarks for AVLMs (Table from general VQA (Agrawal et al. knowledgo-aware KVQA (Shah et al. 2019) reflects increasing demands for external knowledge. OK-VQA
5). The progression
2015) to fact-based FVQA (Wang et al. 2016 Lin et al. 2023cb and
(Marino et al., 2019) and its successor
(Reichman et al. 2023 have become standard benchmarks requir-
ing knowledge beyond visual content. Specialized variants target encyclopedic knowledge (Mensink et al.,
2024). named entities (Lerner et al.) 2022 Qiu et al. 2024), and synthetic knowledge generation ([uu et al.
2023), cultural domains 4Agarwal et al.;
2024 Advanced reasoning benchmarks like A-OKVQA (Schwenk
et al. 2022 and Visual Riddles (Bitton- (Bitton-Guelta et al.
2024) combine knowledge requirements with com-
plex compositional reasoning, while CRIC (Gao et al.
2019 tests compositional reasoning on vision and
commonsense.
6.1.4 Physical commonsense and affordances
VLMs often lack robust understanding of physical propertics and object affordances visible in scenes, com- monly misunderstanding material properties, physical constraints, or predicted behavior under manipulation (Yi et al., 2019 Wang et al. 2024d). This gap necessitates specialized bridging mechanisms that translate between continuous visual features and discrete symbolic representations.
Two augmentation routes have proven effective. First, symbolic vector or graph representations bridge continuous perception and discrete physical predicates. Vector Symbolic Architectures (Montone et al. 2017 Kovalev et al. 2021 represent spatial relationships as high-dimensional vectors supporting symbolic operations like binding and unbinding within the neural architecture itsclf. Graph neural networks operat- ing on coupled scene-concept graphs (Dong et al., 2024; [ong et al., 2023) enable cross-modal knowledge exchange, with Concept.et integration showing 8- 8-12% improvements
tasks requiring physical common- sense reasoning. Second, tool-enabled pipelines call physics or geometry modules (depth estimation, meshing, simple simulators) to test feasibility measure quantities before answering. The structured nature of ex- ternal knowledge allows models to access explicit relationships like "glass IsA fragile' or 'liquid HasProperty flows howss_dwwwwwrr that may not be reliably encoded in learned parameters.
Physical reasoning evaluation is less explicitly represented in current benchmarks but appears implicitly across multiple datasets (Table 5 The Compositional 4D dataset (Wang et al. 2024d) specifically targets understanding of physical dynamics and material properties across time. Visual Commonsense Reasoning (VCR) [Zellers et al.] 2018) requires understanding of physical plausibility and social dynamics, while ex- plainable ble reasoning benchmarks (Cao et al.) 2019) often involve physical world knowledge. The relative scarcity of dodicated physical commonsense benchmarks represents a gap in current evaluation frameworks, despite this being a critical limitation of VLMs.
6.1.5 Action and embodiment
In robotics and embodied AI applications, the challenge of translating visual scenes and natural language instructions into executable action sequences requires precision and verifiability that standalone VLMS cannot
12

---

Published in Transactions on Machine Learning Research (01/2026)
provide. Angmentation patterns treat VLMs as interpreters that generate symbolic action specifications for downstream execution.
Methods like ViLaln (Shirai et al. 2023 generate PDDL specifications that symbolic planners execute with guaranteed optimality properties. Systems employing Answer Set Programming (Zhang et al., 2022b) or other formal planning languages leverage logical consistency checking. Many approaches implement plan execute observe reflect loops that re-plan on mismatch, with retrieval supplying braries and environment maps. Tool interfaces expose planners and low-level skills through well-defined APIs, while safety and validity enforced through logic layers checking preconditions, effects, and con- straints. This vision-to-symbol translation separates flexible perception from rigorous planning, enabling reliable task execution in physical enviromments.
li-
Embodied AI benchmarks evaluate AVLMs' ability to translate understanding into executable actions (Table 5. Web-based environments like WebArena (Zhou et al. 2023) and ScrcenAgent (Niu al. 2024) test agents realistic computer control tasks, while Spider2-V V Cao et al. 2024) focuses on data science workflows. Robotic manipulation benchmarks (Gao et al. 2023b) evaluate physical grounding and tool use in real-world settings. These ggent-focused evaluations represent shift from passive -assweinn to active, goal- directed interaction with environments, requiring integration of perception, reasoning, and action execution.
6.2 Common Architectural Patterns in AVLMs
While the surveyed papers employ diverse implementation strategies, our analysis reveals three fundamental architectural patterns that have emerged as dominant paradigms for augmenting vision-languag, language models. These patterns represent crystallized design solutions that address specific computational challenges in vision- language understanding. These patterns distribute differently across fusion stages, with retrieval-based approaches primarily operating at the input stage, while symbolic computation patterns span both middle and late fusion paradigms.
6.2.1 The Retrieval- Integration Pipeline Pattern
The retrieval-reasoning pipeline has evolved from simple keyword matching to sophisticated neural retrieval systems. The evolution of this pattern over the past decade (2016-2024) reflects increasing sophistication how external knowledge indexed, accessed and integrated in the following ways:
Early Simple Retrieval (2016-2018): Direct keyword matching or TF-IDF based retrioval from
knowledge bases, with retrieved facts concatenated prompts (Wang et al. 2015 Narasimhan &
Schwing. 2018
Learned Dense Retrieval (2019-2020): Introduction of learned emboddings using pretrained
encoders like CLIP, enabling semantic similarity search (Li et al. 2020 Zhang et. al. 2020
Joint VIM-Retriever Training (2021-2022): End-to-end training of retriever and VLM com- ponents, optimizing retrieval for downstream task performance
(Chnn et al.) 2022bb Gui et al. 2021)
Tool-Augments Retrieval (2023-2024): Retrieval as one tool among many, with VLMs learn-
ing when and what to retrieve (aan & Xie 2024 Hao et al. 2024b
Critical design decisions in this pattern include the choice of embedding space, where CLIP- retrieval demonstrates strong performance on vision-language tasks (Gui et al., (002), retricval granularity with
sentence-level retrieval showing advantages for factoid questions (Chen et al. 2022b), and integration mech-
anism where attention-based fusion consistently out perforrsms simple concatenation Yuuan el. al. 2023b Weng
et al. 2024
6.2.2 The Intermediary Program Pattern
This pattern treats program synthesis as a bridge between neural perception and symbolic computation, with the VLM generating executable code that operates on structured representations invokes external
13

---

Published in Transactions on Machine Learning Research (01/2026)
tools. The generated programs interpretable reasoning chains that can be verified, debugged, and modified. Importantly, we distinguish between domain-specific program synthesis and general-purpose code generation, as they represent different points on the expressiennss-tractane
spectrum.
This pattern's evolution has followed a similar progression to the vetrieval-reasonie pipeline, moving from static patterns dynamic, learned behaviors:
Static Logic VLM Outputs (2018-2019): Early approaches applied predefined symbolic
logic engines (ASP. Prolog) to VLM-cxtracted scene graphs, requiring manual rule specification
[Aditya et al.
2018 Riley & Sridharan 2019)
Domain-Speccfic Program Syuthesis (2020-2022): VLMs learned to generate programs in
constrained domain-specific languages (DSLs) with guaranteed executability. VQA (Yi et al. 2018) synthesizes functional programs over fixed set of visual primitives, while Zhang et al. (2025a) generates first-order logic expressions
Gereral-Purpose Code Generation (2022-2023): Shift to generating Python SQL code with
broader expressiveness but without execution guarantees. ViperGPT (Surís et al. 2023) generates unrestricted Python code composing vision APIs, while Gupta & Kembhavi programs for visual reasoning
2022 produces Python
Dynamic Tool Orchestration (2023-2024): VLMs as orchestrators selecting and composing
heterogencous tools including APIs, specialized models, and code exccution environments (Hu al.
2023c Wu al.
2024b
Liu al. 2023b
The key architectural components include: (1) Program specification language DSLs offer tractability with limited expressiveness (NS-VQA's 20 primitives achieve 99.8% on CLEVR (Yi et al. 2018)), ), while
general-purpose languages enable broader capabilities but require error handling; (2) Execution euvironment ranging from symbolic execulors for DSLs to sandboxed Python interpreters with vision library access; (3) Error handling mechanisms recent approaches like Mishra al. (2024) incorporate execution feedback for iterative refinement. ViperGPT demonstrates the gencral-purpose approach's 's flexibility, achieving strong performance across diverse visual reasoning tasks through unrestricted Python generation (urís et al. 2023
6.2.3 The Graph Fusion Pattern
Graph-mediated fusion explicitly models relationships between visual elements and external knowledge through graph structures, enabling structured reasoning over combined perceptual and symbolic informa- tion. This pattern typically involves three stages: graph construction (from visual input and/or external knowledge), graph alignment (connecting visual and symbolic graphs), and graph neural network processing for joint reasoning.
Key Design Variations:
Scene- Concept Graph Coupling: Methods like VQA-GNN (Wang et al, 2022c and MAIL (Dong
ct al. 2024) construct parallel scene graphs (from images) and concept graphs (from knowledge bases), using shared entities bridges. Graph attention networks enable cross-modal message passing while preserving intra-modal structure, with VQA-GNN showing 4.6% improvement
GQA through this approach (Wang et al. 2022ch
Multimodal Knowledge Graph Integration: Approaches incorporating MMKGs X et al. 2021 where nodes contain both visual exemplars and textual descriptions, enabling richer cross-modke grounding through joint embedding spaces
2024 Liu et al.
Dynamic Graph Construction: Methods that construct query-specific subgraphs rather than us- ing fixed graph structures, with Li et al. (2022b) demonstrating improved efficiency through adaptive graph pruning.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Critical implementation choices include graph representation (heterogencous vs. homogeneous nodes), align- ment mechanisms (entity matching vs. learned attention), and message passing strategies (synchronous vs. asynchronous updates). The graph-mediated pattern provides a principled way to preserve structural infor- mation while enabling neural reasoning, making it particularly effective for tasks requiring explicit relational understanding ween visual elements and conceptual knowledge.
6.2.4 Cross-Pattern Observations
Several key insights emerge [rom analyzing these patterns:
1. Temporal Distribution: Retrieval-based approaches concentrate in early fusion (where they mod- ify inputs), while program and graph-based patterns distribute across middle and late fusion stages (where they can interact with intermediate representalions or refine oulputs).
2. Complementarity: The most successful recent systems combine patterns - for example, using retrieval to gather relevant facts, then applying program synthesis for precise computation over retrieved information (Castrejon et al. 2024 Lu et al. 2023
3. eeteppretabiltty-Simpiio Trade-off: Program-based approaches offer highest interpretability through executable traces but require more specialized, task-specific training. Graph- Graph-based methods provide moderate interpretability through explicit relational structure. Pure retrieval offers limited interpretability but is simplest to implement.
4. Scalability Characteristics: Retrieval scales well with knowledge base size using approximate nearest neighbor search (Chen et al. 2022b), program synthesis complexity grows with the size of the DSL or API set (Yi et al. 2018), while graph methods face quadratic complexity in number of nodes, requiring approximation techniques for large graphs (Wang et al. 2022c
These architectural patterns provide a technical foundation for designing augmented vision-languse systems, with the choice of pattern depending on task roquirements for accuracy, interprotability, and computational efficiency.
6.3 Future Directions for Iision-Centril Augmentation Research
Cross-Modal Knowledge Integration Through Bidirectional Grounding: Rather than treating re- trieved information as passive context, future research should explore augmentation techniques that leverage cross-modal correspondences more deeply. For example, retrieved knowledge about typical spatial layouts (e.g., "monitors are usually on desks') could constrain visual parsing, while visual evidence could trigger tar- geted knowledge retrieval. Multimodal knowledge graphs where nodes represent visual concepts with both image exemplars and textual descriptions (Jiang & Meng 2023; Xi et al. 2024 could enable richer cross- modal reasoning through joint embedding spaces. This bidirectional interaction between symbolic knowledge and visual processing could improve both grounding accuracy and reasoning efficiency by ensuring consis- tency between what the model sees and what external knowledge suggests should be present.
Embodied Vision-Language Systems with Reinforcement Learning for Tool Use: The integration of VLMS with physical embodiment and interactive environments presents unique augmentation opportuni- ties beyond one-shot prediction. Embodied systems can iteratively refine understanding through interaction: moving cameras to viewpoints, manipulating objects to reveal hidden properties, or executing actions to test hypotheses about the physical world (Gao et al. 2023b). Recent advances in reinforcement learn- ing for visual tool use, exemplified by approaches like VTool-R1 (Wu et al.] 2025). demonstrate how RL can train VLMs to dynamically select and compose vision lools based on environmental feedback and lask requirements. Future research should develop reward functions that balance exploration (trying new tool combinations) with exploitation (using known effective strategics), while incorporating human preferences ensure safe and interpretable tool usage patterns. This includes learning when request human intervention for ambiguous visual scenes safety-critical decisions in autonomous systems.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Interpretable Visual Program Synthesis for Safety-Critical Applications: While current visual pro-
gramming approaches demonstrate success on controlled benchmarks (Surís et al. 2023 Gupta & Kembhavi, 2022), deploying these systems in safety-critical domains requires advances that combine program synthesis with interpretability mochanisms. Future systems must handle partial observability through techniques like uncertainly-aware program generation that explicitly represents sonfidence in different execution paths. (Bao ct al. 2023 Vedantam et al. 2019; Chac & Kim 2022) The challenge of providing interpretability becomes paramount in high-stakes applications. Systems should produce visual reasoning chains that show which image regions were examined, what external tools were invoked, and how intermediate results combined to reach conclusions. This requires developing visualization techniques that render program execution traces overlaid on images with attention heatmaps, tool call annotations, and confidence scores. For medical imag- ing, autonomous driving, or industrial inspection, these interpretable execution traces serve dual purposes, enabling domain experts to verify correctness and providing auditable records for regulatory compliance. The key is balancing completeness (showing all reasoning steps for full transparency) with comprehensi- bility (avoiding overwhelming users with excessive detail through hierarchical or interactive visualization approaches).
Scalable Visual Program Libraries and Transfer Learning: Current visual programming approaches often require task-specific program templates or limited APIs. Future research should develop methods for building compositional program libraries that grow through experience, enabling VLMs to synthesize increas- ingly complex vision pipelines by combining learned subroutines. This includes meta-learning approaches that discover reusable visual reasoning patterns across tasks and transfer learning techniques that adapt programs from source domains (where supervision is abundant) to target domains (where it is scarce). The vision-language community would benefit from standardized APIs and benchmark tasks for visual program synthesis, analogous to how HuggingFace standardized model interfaces for NLP.
Adaptive Augmentation Based on Task Uncertainty: Rather than applying fixed augmentation strate- gies, future AVLMs should dynamically determine when and how to invoke external resources based on un- cerlainly estimation. For instance, a model confident its counting ability for sparse scenes might bypass external tools, while requesting symbolic computation for cluttered environments. This adaptive approach roquires developing calibrated uncertainty measures for different aspects of visual reasoning (spatial rela- tionships, object recognition, attribute prediction) and learning policies that optimize the trade-off between accuracy gains and computational costs of augmentation.
7 Conclusion
Vision-Language Models have revolutionized Al's ability to connect vision and language, yet standalone models struggle with factual accuracy, complex reasoning, adaptability, and interprctability. This system- atic review charted the landscape of Augmented Vision-Language Models (AVLMs), which overcome these limitations by integrating VLMs with external symbolic information systems and computational tools. We surveyed a diverse range of techniques, categorizing them by fusion timing (early, middle, late) and the nature of augmentation (retrieval, symbolic computation, combined), revealing clear consensus: augmenting VLMs significantly boosts performance and interpretability on knowledge-intensive and reasoning-heavy tasks by synergizing neural pattern recognition with symbolic precision (Marino et al. 2020 Vedantam et al. 2019 Bitton-Guetta et al. 2024 Yan & Xie 2024 Hu et al. 2023ch. A particularly powerful paradigm emerging from this landscape is tool use, which offers a flexible and unifying abstraction for AVLM design. This ap- proach frames the VLM as an intelligent orchestrator trained to select and utilizo external capabilities (such as knowledge bases, calculators, code execution, specialized algorithms, formal reasoners) encapsulated as 'tools," * enabling modularity and scalability. Significant challenges remain in managing interaction complex- ity, ensuring scalability and efficiency, guaranteeing robustness against unreliable external inputs, developing comprchensive evaluation methods, and refining the tool integration mechanisms themsclves. Nevertheless, the advaeementt of AVLMs, particularly through the lens of tool use, represents a crucial progression towards more capable, reliable, and trustworthy Al systems that effectively blend neural perception with symbolic reasoning, allowing them to not only see and describe the world but also reason about it with greater depth, accuracy, and transparency.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Methodology
This section describes the process of gathering relevant articles for this survey, following the Preferred Reporting Items for Systematic reviews and Meta-Ana Analyses (PRISMA) guidelines. The goal of this approach is to avoid bias when selecting what papers to review, focusing on the merits of the paper and the relevancy to the topic of AVLMs.
A.1 Search Strategy
A.1.1 Databases and Search Queries
We utilized two primary databases for our literature search:
Google Scholar: Known for its extensive coverage of scholarly publications across disciplines.
Semantic Scholar: Provides advanced search capabilities and citation analysis, facilitating the identification of semantically relevant works.
A.1.2 Search Terms
We formulated specific search queries to capture studies related to augmented vision-language models inter- acting with symbolic systems during inference. The search strategy used the strengths of both databases by employing an iterative process of testing and refining the search query until the resulting set of papers was adequately relevant. Google Scholar is more sensitive to the inclusion of keywords, and so we used a combination Boolean operators to refine the results eflectively.
The search query used in Google Scholar was:
("augmented" OR "knowledge" "knoxledge graphs" OR "knowledge augmentation" OR "commonsense knowledge" OR "commonsense reasoning" OR "tool use" OR "retrieval augmented" OR "retrieval-augmente" "external knowledge" OR "neural symbolic" OR "neural-symbolic" OR "symbolic") AND
OR
("vision-language" OR "vision language" OR "visual question answering" OR "image question answering" OR "video question answering" OR "image caption" OR "video caption" OR "image text" OR "spatial reasoning" OR "visual reasoning")
AND
("neural network" OR "machine learning" OR "artificial intelligence" "deep learning") -"virtual reality" -"augmented reality"
Semantic Scholar is less sensitive to keywords and more of semantic search, so for this database, we employed set of targeted queries to capture key aspects of our research focus:
"Commonsense reasoning in visual question answering"
"nowledgge graphs for image or video captioning"
'External knowledge visual reasoning'
NNurra-symmbllc vision-language models"
'Tool use in vision-language tasks"
17

---

Published in Transactions on Machine Learning Research (01/2026)
Rettrieallauggmented image question answering'
'Symbolic reasoning in Al for vision"
"Commonsense in image-text models'
Neurol-symbolic visual question answering"
MMltimodal knowledge graph LLM"
A.2
Inclusion and Exclusion Criteria
To ensure the relevance and quality of the studies included in this review, we established clear inclusion and exclusion criteria.
A.2.1
Inclusion Criteria
Relevance: Studies that describe machine learning models integrating external symbolic informa- tion systems during inference.
Language: Publications written in English.
Implementation Focus: Papers providing detailed descriptions of implementation methods rather than purely conceptual theoretical discussions.
Vision-Language Tasks: Research focusing on tasks such as visual question answering, image
captioning, and video captioning where the input is imagery and/or text and the output is natural language text.
A.2.2 Exclusion Criteria
We excluded studies that did not align with the focus of this review, such as:
Prompting Techniques: Research solely prompt enginecring techniques that rely internal reasoning pallerns without external data augmentation (e.g., chain-offthought prompling).
SellProomping/Reeusiv Prompting: Methods that involve iterative querying without inte- gration of external symbolic information systems.
Synthetic Data Generation: Studies focusing on generating synthetic data to improve model performance without external symbolic system interaction.
Architectural Modifications Without External Integration: Papers discussing model archi- tectures like vision encoder adapters for large language models that do not involve external symbolic systems during inference.
Training with Structured Knowledge: Research that involves training models with external knowledge but does not allow for the external knowledge to be modified or read during inference (e.g,, methods where external knowledge is embedded in model parameters).
A.3
Selection Process
The selection process involved several iterative sleps to refine and identify the most relevant studies.
A.3.1
Initial Search Results
The search yielded 980 papers after filtering by category and removing irrelevant results based on titles and abstracts.
Google Scholar:
Semantic Scholar: The targeted queries returned 1,332 papers.
18

---

Published in Transactions on Machine Learning Research (01/2026)
A.3.2
Total Papers Collected
In total, 2,312 papers were collected from both databases.
A.3.3 Relevance Scoring
In aligmment with the theme of augmented models, we utilized the GPT-4o OpenAl model (gpt-4o- 2024-08-06) to assist the relevance asscssment:
Automated Categorization: GPT-4o was prompted to categorize each paper and assign a rele- vance score ranging from to 10 based on the alignment with the review topic.
Threshold for Inclusion: Papers scoring less than out of 10 were excluded from further consideration.
Iteration and Validation: The relevance scoring process was iterated, and we ensured that all highly relevant papers were retained, even if they narrowly missed the initial threshold.
A.3.4 Manual Screening
Total Papers After Automated Filtering: 616 papers remained after applying the relevance threshold.
Full-Text Assessment:
We conducted a thorough manual review of the full text of these papers.
Final Selection: After removing duplicates and papers not meoting the inclusion criteria, 264 papers selected for detailed analysis. See Figure 7
Records identified through (n 2312)
Other sources = 7)
Records after
deduplication (n=2277)
Records screened by GPT4o (n=2277)
Records excluded (n=1696)
Full-text articles eligibility (N:581)
Full-text articles excluded, with reasons (n=317):
Out of scope (n=292)
No full text (nw1)
Foreign language (n=1)
Studies included (n=264)
Duplicate (n=20)
Lack quality (n=3)
Figure 7: PRISMA Flowchart
A.4 Data Extraction and Synthesis
From the selected studies, we extracted pertinent information to facilitate a comprehensive understanding of the methods:
Integration Techniques: Description of how external systems were integrated with vision-language models, classified into early fusion, middle fusion, and late fusion methods.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Categorization of external symbolic systems used, such as knowledge graphs, symbolic logic engines, and program synthesis tools.
Types of External Symbolic Systems:
Tasks Addressed: Identification of the specific vision-language tasks tackled by each study, in- cluding visual question answering, image captioning, and others.
Implementation Details: Detailed examination of the models' architectures, including the inter- action mechanisms with external symbolic systems during inference.
A.5 Quality Assessment
We assessed the quality of the included studies based on:
Clarity of Methodology: Transparency and reproducibility of the methods described.
Experimental Rigour: Adequacy of experimental design, including dalaset usage, evaluation protocols, and statistical significance of results.
Contribution to the Field: The extent to which the study advanced understanding or provided innovative solutions augmented vision-language models.
A.6 Limitations
While we aimed for comprehensive review, certain limitations exist:
Unpublished works or those not indexed in the selected databases may have been missed.
Publication Bias:
Language Restriction: English publications were excluded, which may omit relevant research conducted in other languages.
Dynamic Field: Given the rapidly evolving nature of machine learning research, new studies may have emerged after the completion of our scarch.
AI Bias: The use of GPT40 filtering papers could potentially remove relevant search results.
By following this systematic approach, we ensured thorough and unbiased selection of relevant literature, providing solid foundation for the subsequent analysis and discussion in this review.
B Categorization Tables
This section contains the tables categorizing the surveyed papers based the fusion method (Early, Middle, Late) and the type of augmentation (Retrieval, Symbolic Computation, Combined). It also includes table summarizing relevant datascts. These tables correspond to the synthesis presented in the main Results section.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Early Fusion Methods in Vision-Language Model Angmentation. Bracketed tags denote pre-
dominant reasoning domain categories: Spatial, Temporal, Knowledge grounding, Physical commonsense, Eiooon/mmmddieent
Retrieval
Prompt Augmentation
Querying KG
Retrieval Encoders
Image Caption
Retrieval FT
Prompt Augmentation
Subgraph Enc
Gao et
2022a
[
2024
[K]
2018
[S
Pabian ot al.
2023
Kan et al
2023
[
(Ravi et al.
2022
K
(Sharifymon manghdddam
ct al.
2024)
[K]
Ranjit a.
2023
[K]
Narasimhan & Schwing
2018
[
Li et
2020
[
[Ghosal a..
2023
[
Rao et
2023
[K]
Khademm et. a.
2023
[K]
1uu al.
2024
K
Vickers al.
2021
[
Zhang s
2020
K
Tiu ct al.
2023
[K]
Liu al.
2124
[
Giuo ct al.
2022
[
Lee & Kim
2021
[K]
Dey ct al.
2021
[K]
Yan Xic
202-
[K]
Wang al.
2015
[S,K]
Li ct al.
2022a
[K]
Xu ot
2024a
[K]
Natu ot
2023
[K]
Lin ot a.. 2022
[[]
Khaliq
2122
[K]
Thalani
2024
[
Lin et al.
2023a
[K]
Yu cl. al.
201991
[S.K]
Torino el al. 21220
[S
Ja Liu eb 2t 2024
[K]
Kuman et
[K]
Barezi Kord jarmshie
2024
[K]
Wang et ml 2022
[K]
Wen
2024
[K]
Zhang cl
2024
[K]
Mogadala
2019 [
Qu et al.
2020
Lin &r Byrne
22122
[
Iscett ct al.
2023
[K]
Zhang al.
20233
[
Pachi et al. (2224)
K
Josht et al. 20/24
[
Wang et al.
2023
K
Garcia-Olano et al.
2021
[
Shevchenko al. 22021 [K]
Salemi
Luo al.
2021
20236
[
Chen eb al 20226
Gur eb al
202T
K
[
Chen el al. 2022c
Ogawa et al.
2024
[
T
Gardéres el.
Jing et al.
2023) [
21120
[
Vo
2022
[
Cui et al
2024
[
Kan et al.
2021
S.K]
Mondal 20244
Zhu et al
2023
[
Gan et a.
20/33
[K]
Hao
2024b
[
Lee 21244 [
Gmil 2021
K
Hao et al.
2024a
[
Yang et al. 2019
[T,K]
Chen al
2021a
[K]
Liang a.
2021
Lerner et
[
Retrieval
Retrieval Encoders (Contimued)
KG Encoding
Encoder Architectures
KG Conv
MMKG Attn
Attention
LSTM
(Chen al.
2021b [
Zhacelard
Lecué
2020
K
Yuan et al.
2023b
K
Yuu et
2020
K
[Zhu et al.
2020b
[
Jiang & Meng
2023
[K]
Weng et al
2024
K
Wu et
2016
[S
Chen et al.
2122a
K
Hussain et al.
2022
[
Salemi al.
2023a
K
Li &r Moens
2022
[
Yc ct al.
2021
Retrieval
Symbolic
Retrieval Encoders (cont.)
Memory
Symbolic
Hu et al.
2022 K
Potapov et. al.
2019 S
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Middle Fusion Methods in Vision-Lanuuage Model Augmentation. Bracketed tags denote pre-
dominant reasoning domain categorics: Spatial, Temporal, Knowledge grounding, Physical commonsense, Action Eebbodmnnndi
Retrieval
Dense Retricval
Graph
Cancept/Scene Fusion
KG Prompt Augmentation
KG/NN Similarity
Li et
2017
[
Li cb al.
20236
[
Zhenng al.
2021
[
Yang et al.
2023
S
(aang et al.
20226
[
151 et. al.
2018
K
Khan el al.
2022h
S.K
Narasimhan
Zhang et al.
aa.
2023c
[2018
[
Khan et
Zhu 2022
2022a
[S.K]
S.K]
Singh et al.
2019
[K]
Wen Peng
|201
[S,K]
Jiang et al.
2020
[T.A,K]
Wu et al.
2024a
K
Wang et al.
2022b
[
Yu et al.
2023
[
Chac & Kim
2022
[K]
Song et. al..
Lt at. |2.22
2023
A,
[KPP]
Lin et al.
20236
K
Du
2022
K
Li al. 2019
K
Zhang et al.,
2022
[
Jia et al.
2023
K
Li et al.
2024s
K
ming Xian et al.
2023
[
Y et
2023
K
Marino et al.
2020
[
Dong al.
2024
S.K
Zhu et al.
Ma ct al
2020a
2022
S
K
Gao ct at.
Zhang ct
|2022
22023a
K
[Xu et al..
2021
T,K]
Cao cl a
Zheng ct a..
Li ct al.
2022b
2019
A,
S S
K
Hou al.
[ cl ai.
20244b
20020
[T,K]
[K]
Wei eb al. 22022
K
Gu ct al.
2019
[S,K]
Narayanan et al.
2021
[K]
Retrieval
Symbolic Computation
Graph
RL
Program Synthcsis
MMKGs
Hypergraphs
(hang et al.
2022c
K
Zhang et al.
2023e
[
Xi et al.
2024
TTKK]
Hu et al.
2023
S
Shi
2022
[SKK]
Hoo ct al.
2022
K
Shurai ct al.
2023
[A,S]
Santicstoban
2024
K
Bougic al. 2018
[A,K]
Wang et al.
2024b
[K]
Zhang ct a.. 2023b
[A]
Ouyang 2024
Li et al.
2021
[
Liu al.
202T
Mishra et al
2024
[K]
Xue ct al.
2024
[
Symbolic Computation
Logic Engines
VSA
Symbolic Graph Ops
Tool Use
7Zhang el al.
2025a
K
Riley & Sridharan
2019
[
Li et al. 2022c
[
Mitchener et al.
202T
[
Liang et al.
2020
[S,K]
2024
S
[satson et al.
2021
K
Wu et al. 2123
[TK]]
Zhang ot a.. 20226
A
Montone et al.
2017
[S]
Zhao 2015
IP.5
[,S
20236
2024
A
C h1o1 et al. 2024
T T,K]
Kovaley et al.
2021
[S]
Yang et a..
2020
[S]
2023c
Li et a.. 2023c
[
Zhang et al.
2023t
[
Wu
2024b
Li ct a... 2023d
K
Hudson & Manning
2019
[S]
Huang ct al. 2021
[K]
CCao et al.
2021
[S,K
Zhang et al.
2023d
[A]
Symbolic Computation
Combined Retr & Symb
Self Play
Agents
Other
(Bebbe et al
2015
[
Aditya
Niu et al.
2024 [A]
Addiya 27
K
Castrejon oon n..
[
Aditya Baral
Misiunas al. 2024 [
Li et al.
2023
Tan et at. 20z
[A
21123
21125a
Xi Nf ml.
21244 h
A,K
Stammeer of al 12112 K
Vitlig
Vatashey Ulimma
[S,K]
Gan of Al
S
22

---

Published in Transactions on Machine Learning Research (01/2026)
Table Late Fusion Methods Vision-Tanguage Model Augmentation. dominant reasoning domain categories: Spatial, Temporal, Knowledge grounding, Physical commonsense,
Bracketed tags denote pre-
Action/Embodinend Embodiment
Retrieval
Symbolic Computation
Dense
Knowledge Graph
Program Synth
Symbolic Engines
Vedantam al.
2019
[S]
et
2018
Sethuraman et
2021
[S,K]
Song al.
2022a [
Cao al.
2022b)
[S.K]
Suris ct
2023
|.
Aditya al
2018
[S]
Song 20226
K
Huang
2020
[K]
Khandelwal al
2023
[S]
Ditor ct 202
[S
2024
K
2022
[S
Subramanian
2123
S.K
Fitter
2021
Guppa Kembhav
[S]
Cunnington
2024
[K]
Bhaisaheb cl
K
Symbolic Computation
Combined
Symbolic Graph Ops
Tool Use
Other
Combined
Li et al.
2023a [A,
Yuan et al.
2023a
K
Xu et al. 2022 [A]
ZZhan al.
2021
[S]
Cesista et al.
2024
[
Singh
2018
K
Sachan
2020
K
Sagur Narasimaan
2020
[S,K]
Cesista
2024
K
Bao
2023
K
Basu ct
2020
[S,K]
Johnston 2023
[]
Zhang
2023
[
Verheyen ct
2023
[TKK]
Table 5: Datasets Relevant to Augmented Vision-Language Models.
Bracketed lags denole predom-
inant reasoning domain categories: Spatial, Temporal, Knowledge grounding, Physical commonsense, Action EEmboninnmddicnnt
Spatial Reasoning [S]
Knowledge Based VQA [
Reasoning VQA [S,K]
CLEVER
Scene Graph
KBVQA
Reasoning VQA
Agrawal et al.
2015
Wang et al.
2016
Lin et al.
2023c
Shah et al. 2019
Marino a.. 2019
Johnson et al. 2016)
S
Schwenk et al. 2022
Reichman et al. 2023
Yi et al. 2019
Gao et al. 2019
Krishna al. 2016
202- Su al.
[i et al. 2022c
7el Zellers et a.
20TS
Shen et al. 2024
Mensink et a.l] 2023
Abraham et al. 2024
Cao et al. 2019
Jain al. 202
Wang al.
2024d
Setton-GGeetaa al.
2024
Cao et al. 2020
Sung et al. 2022
A Agarwal el 2024
(Qiu et al. 2024
Lerner et a..
2022
Knowledge and Spatial [K,S]
Agents [A]
Task Specific
Knowledge and Spatial
Agents
Robotics [A]
Other (Task) [K]
Chen et al.
2023
Niu et al.
2024
Hayashi et al.
2024
Wang et al. 2024a
Zhou el al.
2023
Gao et al,
2023b
Jin et al
2024
Zhang et al.
2023a
Cao el
2024
Hu et al
2023a
23

---

Published in Transactions on Machine Learning Research (01/2026)
C
Glossary of Key Terms
Table 6: Key Terms and Definitions
Term
Definition
Vision-Language
Model
Model jointly processing visual and textual modalities through tasks like VQA, captioning, or image-text retrieval.
(VTM)
Augmented VLM (AVLM)
VLM integrated with external symbolic systems, APIs, tools during in- ference to overcome standalone limitations.
Neural-Symbolic System
Hybrid architecture combining neural pattern recognition with symbolic log- ical reasoning and knowledge representation.
Early Fusion
Integration external information at input stage, before VLM internal pro- cessing begins.
Middle Fusion
Integration during VLM's inference, interacting with intermediate represen- tations before final output.
Late Fusion
Integration after VLM generates initial output, typically for validation, re- finement, or explanation.
Retrieval Augmented Gener- ation (RAG)
Retrieving relevant external information to provide as context for model generation.
Knowledge Graph (KG)
Structured knowledge as directed graph with entity nodes and relationship edges, encoded as triplets.
Scene Graph
Structured visual scene representation with object nodes and spatial rela- tionship edges.
Program Synthesis
Automatic generation of executable code (Python, SQL) by models for rea- soning operations.
Tool Use
VLMs dynamically invoking external tools (calculators, APls, vision mod- ules) based on task needs.
Graph Neural (GNN)
Network
Neural architecture for graph data, enabling message passing across nodes and edges.
Dense Retrieval
A retrieval method using learned dense vector embeddings to find semanti- cally similar content through vector similarity metrics rather than keyword matching.
Visual Question Answering (VQA)
Task requiring natural language answers to questions about visual content.
ConceptNet
Multilingual common-sense knowledge graph with semantic concept net- works.
Answer Set Programming (ASP)
Declarative programming paradigm for knowledge representation and logical constraint solving.
PDDL
Planning Domain Definition Language standardized language for express- ing planning problems and domains in automated planning systems.
References
Savitha Sam Abraham, Marjan Alirezaic, and L. D. Racdt. Clevr-poc: Reasoning-intensive visual question answering in partially observable environments. pp. 3297 3313, 2024.
Somak Aditya. Explainable image understanding using vision and reasoning. pp. 5028 5029, 2017.
Somak Aditya and Chitta Baral. Deepiu: An architecture for image understanding. 2016.
24

---

Published in Transactions on Machine Learning Research (01/2026)
Somak Aditya, Yezhou Yang, Chitta Baral, and Yiannis Aloimonos. Answering image riddles using vision and reasoning through probabilistic soft logic. arxiv preprint, arXiv:1611.05896v1 2016.
Somak Aditya, Rudra Saha, Yezhou Yang, and Chitta Baral. Spatial knowledge distillation to aid visual reasoning. arXie preprint, arXiv:1812.0363112, 2018.
Somak Aditya, Yezhou Yang, and Chitta Baral. Integrating knowledge and reasoning in image understanding. IJCAI 2019, 2019.
Pulkit Agarwal, S. Sravanthi, and Pushpak Bhattacharyya. Indifoodvqa: Advancing visual question answer- ing and reasoning with a knowledge-infused synthetic data generation pipeline. pp. 1158-1176, 2024.
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering. arXiv prepriat, arXiv:1505.006687,, 2015.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Mioch, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadch, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, An- drew Zisserman, and Karen Simonyan. Flamingo: & visual language model for few-shot learning, 2022. 14198
URL httts:://arttii http:://aaxivvvooaaa/////aa
Wenbin An, Feng Tian, Jiahao Nie, Wenkai Shi, Haonan Lin, Yan Chen, Qianying Wang, Y. Wu, Guang Dai, and Ping Chen. Knowledge acquisition disentanglement for nnowledge-basedd visual question answering with large language models. ArXiv, abs/2407.15346, 2024.
An Liu, Chenxi Huang, Ning Xu, Hongshuo Tian, J. Liu, and Yongdong Zhang. Counterfactual visual dia- log: Robust commonsense knowledge learning from unbiased training. IEEE Transactions on Multimedia, 26:1639-1651, 2024.
Yajie Bao, Tianwei Xing, and Xun Chen. Confidence-baseed interactable neural-symbolic visual question answering. Neurocompating, 564:126991, 2023.
Elham J. Barezi and Parisa Kordjamshidi. Find the gap: Knowledge base reasoning for visual question
answering. Xiv preprint, arXiv:2404.10226v1 2024.
Kuntal Basu, Farhad Shakerin, and Gopal Gupta. AQuA: ASP-Based Visual Question Answering. In Ekaterina Komendantskaya and Yanhong A. Liu (eds.), Practical Aspects of Declarative Languages (PADL 2020), volume 12007 of Lecture Noles in Compuler Science. Springer, Cham, 2020. doi: 4. URL http::://aal..rr///000000//////3
000007797783000-399000
Ghada Besbes, H. B. Zghal, and H. Ghézala. An ontology-driven visual question-answering framework. 2015 19th International Conference on Information Visualisation, pp. 127 132, 2015.
Shabbirlussain Bhaisaheb, Shubham Paliwal, Rajaswa Patil, Manasi S. Patwardhan, L. Vig, and Gautam M. Shroff. Program synthesis for complex qa on charts via probabilistic grammar based filtered iterative back- translation. pp. 2456 2470, 2023.
Nitzan Bitton-Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. Visual riddles: commonsense and world knowledge challenge for large vision and language models. arXie preprint, arXivv2407.19474x2. 2024.
Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jon Lenchner, Nick Linck, Andrea Loreggia, Keerthiram Murugesan, Nicholas Mattoi, Francesca Rossi, and Biplav Srivastava. Thinking fast and slow in ai. Proceedings of the AAAI Conference on Artificial Intelligence 2021, 35(17), 15042-1504$ 2020.
Nicolas Bougie, Limei Cheng, and R. Ichise. Combining deep reinforcement learning with prior knowledge and reasoning. ACM SIGAPP Applied Compuling Review, 2018.
25

---

Published in Transactions on Machine Learning Research (01/2026)
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, K. Choromanski, Tianli Ding, Danny Driess, Kumar Avinava Dubey, Chelsea Finn, Peter R. Florence, Chuyuan Fu, Montse Gonzalez Are- nas, K. Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian lchter, A. Irpan, Nikhil J. Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, S. Levine, H. Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, M. Ryoo, Grecia Salazar, Pannag R. Sanketi, P. Sermanct, Jaspiar Singh, Anika Singh, Radu Soricut, Huong Tran, Vincent Van- houcke, Q. Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Ted Xiao, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-languager (bss/2307.11811 2023.
models transfer web knowledge to robotic control. ArXiv,
Qingxing Cao, Bailin Lí, Xiaodan Liang, and Liang Lin. Explainable high-order visual question reasoning: A new benchmark and knowledge-routed network. ArXiu, abs/ hbbs/199.10122 2019.
Qingxing Cao, Bailin Li, Xiaodan Liang, Keze Wang, and Liang Lin. Knowledge-ronted visual question reasoning: Challenges for doep representation embedding. JEEE Transactions on Newral Networks and Learning Systems, 33:2758- 2767, 2020.
Qingxing Cao, Wentao Wan, Keze Wang, Xiaodan Liang, and Liang Lin. Linguistically routing capsule network for out-of-distribution visual question answering. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1594-1603, 2021.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, and Tao Yu. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint, arXiv:2407.1095611 2024.
Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, and Jasper Uijlings. Hammr: Hierarchical multimodal react agents for generic vqa. arXiv preprint, arXiv:2404.05465v2, 2024.
Franz Louis Cesista Multimodal structured generation: Cvpr's 2nd mmfmm challenge technical report. arXiv preprint, arXiv:2406.114000 11403,2, 2024.
Franz Louis Cesista, Rui Aguiar, Jason Kim, and Paolo Acilo. Retricval augmented structured generation: Business document information extraction as tool use. arXiv preprint, MXXi:2440.202445 2024.
Jinyeong Chae and Jihie Kim. Uncertainty-aaee visual question answering: Estimating semantic incon- sistency between image and knowledge base 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1-9, 2022.
Kezhen Chen, Qiuyuan Huang, Yonatan Bisk, Daniel J. McDuff, and Jianfeng Gao. Kb-vip: Knowledge based vision and language pretraining. 2021a.
Kezhen Chen, Qiuyuan Huang, Danicl McDuff, Yonatan Bisk, and Jianfeng Gao. Krit: Knowledge- reasoning intelligence in vision-language transformer. 2022a.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Vorga, and William W. Cohon. Murag: Multimodal retrioval- augmented generator for open question answering over images and text. pp. 5570, 2022b.
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang, Can pre-trained vision and language models answer visual information-seeking questions? ArXiv, abs/2302.11713, 2023.
Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z. Pan, Zonggang Yuan, and Huajun Chen. Zero-shot visual question answering using knowledge graph. ArXiv, abs has/2107.0534,, 2021b.
Zhuo Chen, Yufen Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang, Jeff Z. Pan, Ningyu Zhang, and Wen Zhang. Lako: Knowledge-driven visual question answering via late knowledge-to-text injection. Proceedings of the 1 1tth International Joint. Conference on Knowledge Graphs, 2022c.
26

---

Published in Transactions on Machine Learning Research (01/2026)
Minkyu Choi, Harsh Goel, Mohammad Omama, Yunhao Yang, Sahil Shah, and Sandeep Chinchali. Towards neuro-symbolic video understanding. 2024.
Wanqing Cui, Keping Bi, J. Guo, and Xueqi Cheng. More: Multi-modal retrieval augmented generative commonsense reasoning. ArXiu, abs /////2222..333 2024.
Daniel Cunnington, Mark Law, Jorge Lobo, and Alessandra Russo. The role of foundation models in neuro- symbolic learning and reasoning. ArXiv, abs/2442.01188, 2024.
Arka Ujjal Dey, Ernest Valveny, and Gaurav Harit. External knowledge augmented text visual question answering, ArXiv, abs/2108.09711? 09971777, 2021.
Junnan Dong, Qinggang Zhang, Huachi Zhou, D. Zha, Pai Zheng, and Xiao Huang. Modality-aware inte- gration with large language models for sewwldge-basee visual question answering. pp. 2417 2429, 2024.
Qinyi Du, Qingqing Wang, Keqian Li, Jidong Tian, Liqiang Xiao, and Yaohui Jin. Calm: Commen-sense knowledge augmentation for document image understanding. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
Thomas Eiter, N. Higuera, J. Octsch, and Michael Pritz. A confidencc--aseed interface for ncuro-symbolic visual question answering. 2021.
Thomas Eiter, N. Higuera, J. Oetsch, and Michael Pritz. A neuro-symbolic asp pipeline for visual question answering. Theory and Practice of Logic Programming. 22:739 754, 2022.
Zalan Fabian, Zhongqi Miao, Chunyuan Li, Yuanhan Zhang, Ziwei Liu, A. Hern' 'adezz Andrés Montes-Rojas, Rafael S. Escucha, Laura Siabatto, Andr'es Link, Pablo Arbel' aez, R. Dodlia, and Ferres. Multimodal foundation models for zero-shot animal species recognition in camera trap images. ArXiv, 0bs/2331.01164 2023.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. ArXiv, abs/2403.11443 2408.1148,, 2024.
Xingyu Fu, Shenmin Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, A. Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, D. Roth, and Bing Xiang. Generate then select: Open-ended visual question answering guided by world knowledge. ArXiv, abs/2305.18842, 2023.
Jingru Gan, Xinzhe Han, Shuhui Wang, and Qingmiing Huang. Open-set knowledge-based visual question answering with inference paths. ArXiv, abs 0bs/2310.08148, 2023.
Chen Gao, Si Liu, Jinyu Chen, Luting Wang, Qi Wu, Bo Li, and Qi Tian. Room-object entity prompting and reasoning for embodied referring expression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:994-1010, 2023a.
Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. Cric: A vqa dataset for compositional reasoning on vision and commonsense. arXiv preprint, 2XXiv:1908.029223. 2019.
Feng Gao, Q Ping, G. Thattai, Aishwarya N. Reganti, Yingting Wu, and Premkumar Natarajan. Transform- retrieve-generater Natural language-centric outside-knowlledee visual question answering. 2022 IEEE/CVF Conference Compuler Vision and Pallern Recognilion (CVPR), pp. 5057-5067 2022a.
Jensen Gao, Bidipta Sarkar, F. Xia, Ted Xiao, Jiajun WU, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12462 12469, 2023b.
Jingying Gao, A. Blair, and M. Pagnucco. A eymmbolic-nuurl reasoning model for visual question answering. 2023 International Joint Conference Neural Networks (IJCNN), pp. 1-9, 2023c.
27

---

Published in Transactions on Machine Learning Research (01/2026)
Jingying Gao, Alan Blair, and Maurice Pagnucco. Explainable visual question answering via hybrid neural-
logical reasoning. 2024 International Joint Conference on Neural Networks (IJCNN), pp. 1-10, 2024.
Yueqing Gao, Huachun Zhou, Lulu Chen, Yuting Shen, Ce Guo, and Xinyu Zhang. Cross-modal object detection based on a knowledge update. Sensors (Basel, Switzerland), 22, 2022b.
Diego Garcia-Olano, Yasumasa Onoe, and J. Ghosh. Improving and diagnosing knowledge-based visual question answering via entity enhanced knowledge injection. Companion Proceedings of the Web Conference 2022, 2021.
François Gardères, Maryam Ziaeefard, Bapliste Abeloos, and Freddy Lécué. Conceptbert: Concepl-aware representation for visual question answering. In Findings, 2020. URL https://api sematicschooaa.. http:::/ppp..emmatttccoooo.... org/Corpus :2: 226284018
Deepanway Ghosal, Navonil Majumder, Roy Ka-Wei Lee, Rada Mihalcea, and Soujanya Poria. Language guided visual question answering: Elevate your multimodal language model using knowledge-cnriched prompis. pp. 12096 12102,, 2023.
Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction. arXiv preprint, arXiv:1904.00560v1, 2019.
Liangke Gui, Borui Wang, Qiuyuan Huang, A. Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for aabio--and-lagguaee Arxiv, abs/2112.05614, 2021.
Yangyang Guo, Liqiang Nie, Yongkang Wong, Y. Liu, Zhiyong Cheng, and Mohan S. Kankanhalli, A Unified End-to-End Retriever-Readee Framework for Knowledge- VQA. 2022.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositionsl visual reasoning without training, arXiv preprint, arXiv:2211.115599 2022.
Shir Gur, N. Neverova, C. Stauffer, S. Lim, Douwe Kiela, and A. Reiter. Cross-modal retricval augmentation for multi-modal classification. ArXiv, abs/2104.081088, 2021.
Dongze Hao, Jian Jia, Longteng Guo, Qunbo Wang, Te Yang, Yan Li, Yanhua Cheng, Bo Wang, Quan Chen, Han Li, and Jing Liu. Knowledge condensation and reasoning for knowledge-based vqa. ArXin, Abs/2440.10037, 2024a.
Dongze Hao, Qunbo Wang, Longteng Guo, Jie Jiang, and Jing Liu. Self-boosstrapped visual-language model for knowledge selection and question answering. 2024b.
Kazuki Hayashi, t. YusukeSakai, Hidetaka Kamigaito, $. Katsuhiko f. TaroWatanabe, Jean- Baptiste, Xincan Feng, Katsuhiko Hayashi, Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Noa Gareía, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, Teruko Mitamura, A. dataset, D. Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, S. Hambardzauyan, Abhina Tuli, Levon Ghukasyan, Fariz Rahman, Hrant Topchyan, David Isayan, M. Harutyunyan, Tatevik Hakobyan, Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men-sch, Chris Bamford, Devendra Singh, Diego Chaplot, Florian de las Casas, Gianna Bres- sand, Guil laume Lengyel, Lucile Lample, Lélio Renard Saulnier, Lavaud MariieAnne Pierre Lachaux, Teven Stock, Le Scao Thibaut, Thomas Lavril, Timothée Wang, Lacroix William, El Sayed, Mistral, Scott Kushal Kafle, Cohen, Shyamal Anadkat, Red Avila, Igor Babuschkin, S. Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Ir wan Bello, Jake Berdine, Gabriel Bernadett- Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Luia- Brakman, Greg Brockman, Tim Brooks, M. Brundage, Kevin Button, Trevor Cai, Rosic Campbell, Andrew Cann, Brit- Lany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Su- Yuan Chen, Ruby Chen, Jason Chen, Mark Chen, B. Chess, Chester Cho, Hyung Casey Chu, Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Simón Niko Felix, Posada Fishman, Juston Forte, Is abella
28

---

Published in Transactions on Machine Learning Research (01/2026)
Fulford, Leo Gao, Elie Georges, C. Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo- Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross Shixiang, Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, B. Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, I. Kan- itscheider, Nitish Shirish, Tabarak Keskar, Logan Khan, J. Kilpatrick, Wook, Christina Kim, Yongjik Kim, Hendrik Kim, Jamie Kirch-ner, Matt Kiros, Daniel Knight, Kokotajlo Lukasz, A. Kondraciuk, Aris Kondrich, Kyle Kon-stantinidis, Gretchen Kosie, Vishal Krueger, Michael Kuo, Tkai Lampe, Teddy Lan, Jan Lee, Jade Leike, Daniel Leung, Chak Ming Levy, Li Rachel, Molly Lim, Stephanie Lin, Mateusz Lin, Theresa Litwin, Ryan Lopez, Patricia Lowe, Lue Anna, Kim Makanju, S. Malfacini, Todor Man- ning, Yaniv Markov, Bianca Markovski, Katie Martin, Andrew Mayer, Bob Mayne, Scott Mayer McGrew, Christine McKinney, Paul McLcavey, McMillan Jake, David McNeil, Aalok Medina, Jacob Mchta, Luke Menick, Andrey Metz, Pamela Mishchenko, Vinnie Mishkin, Evan Monaco, Daniel Morikawa, Tong Moss- ing, Mira Mu, Oleg Murati, David Murk, Ashvin Mély, Retichiro Nair, Rajeev Nakano, Nayak Arvind, Richard Neelakantan, Hyeonwoo Ngo, Noh Long, Cullen Ouyang, Jakub O'Keefe, Alex Pachocki, J. Paino, Ashley Palermo, Giambat tista Pantuliano, Joel Parascandolo, Emy Parish, Alex Parparita, Mikhail Pas- S08, Andrew Pavlov, Adam Peng, Filipe Perel-man, de Avila Belbute, Michael Peres, Petrov Henrique, Pondé, Michael Oliveira Pinto, Michelle Pokrass, Vitchyr H. Pong, Tolly Pow-ell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ry-der, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Staudacher, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojeicch Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Tianhao Shengjia Zhao, Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, Roozbeh Mottaghi. 2022, Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Fsiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Na- man Goyal, Authony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, Punit Artem Korenev, Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar-tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly-bog, Yixin Nie, Andrew Poulton, Jeremy Reizen-stein, Rashi Rungta, Kalyan Saladi, Liang Wang, Wei Zhao, Zhuoyu Wei, Jingming Liu, SimKGC, Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, J. Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Licheng Yu, Patrick Poirson, Shan Yang, Alexan- der C. Berg, Tamara Berg, Modeling, Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen, Mmmu, Susan Zhang, Stephen Roller, Mikel Arletxe, Shuohui Chen, Christopher De-wan, Mona Diab, Xi Xian Lí, Todor Victoria Lin, Myle Ott, Kurt Shuster, Punit Daniel Simig, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer. 2022a, Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi 2020, Bertscore, Yao Zhang, Haokun Chen, Ahmed Frikha, Yezi Yang, Denis Krompass, and Jindong Gu. Towards artwork explanation in large-scale vision language models. pp. 705-729 729, 2024.
Y. Heo, Eun-Sol Kim, Woo Suk Choi, and Byoung- Zhang. Hypergraph transformer: Weakly-supervied multi-hop reasoning for snwledge-based visual question answering. ArXio, abs/2204. 2022.
P. Hitzler, Md Kamruzzaman Sarker, and Aaron Eberhart. Neuro-symbolic spatio-temporal reasoning. 2022.
Jingyi Hou, Xinxiao Wu, Xiaoxun Zhang, Yayun Qi, Yunde Jia, and Jiebo Luo. Joint commonsense and relation reasoning for image and video captioning. pp. 10980, 2020.
29

---

Published in Transactions on Machine Learning Research (01/2026)
Cheng- Yu Hsich, Sibei Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander J. Ratner, Chen- Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. ArXiu, abs/2308.00675, 2023.
Joy Hsu, Jiayuan Mao, and Jiajun Wu. Ns3d: Neuro-symbolic grounding of 3d objects and relations. 2023 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 2614-2623, 2023.
Xinyue Hu, Lin Gu, Qi A. An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, R. M.
Summers, and Yingying Zhu. Expert knowledge-awee image difference graph representation learning for differenco-aware medical visual question answering. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Dala Mining, 2023a.
Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, K. Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision- language models. 2024 IEEE/CVF Conference on Compuler Vision and Pullern Recognilion (CVPR), pp. 9590-9601, 2023b.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint, ar arXiv:2406.0942
2024.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, C. Schmid, David A. Ross, and A. Fathi. Reveal: Retrieval-augmented visual-language pro-training with multi-source multimodal knowledge memory. 2023 IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 23369-200 23379, 2022.
Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language model agent. arXiv preprind, 2023c.
Feicheng Huang, Zhixin Li, Haiyang Wei, Canlong Zhang, and Huifang Ma. Boost image captioning with knowledge reasoning. arXiv preprint, MxXi:2011.0002770 2020.
Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, M. Naik, Le Song, and X. Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. pp. 25134-25145, 2021.
Zhiao Huang. Feng Chen, Yewen Pu, Chun-Tis Lin, Hao Su, and Chuang Gan. Diffvl: Scaling up soft body manipulation using vision-language driven differentiable physics. ArXie, abs/2312.06408. 2023.
Drew A. Hudson and Christopher D. Manning. Learning by abstraction: The neural state machine. arXiv
preprint, arXiv:1907.03950v2 2019.
Afzaal Hussain, Ifrah Maqsood, M. Shahzad, and M. Fraz. Multimodal knowledge reasoning for enhanced visual question answering. 2022 16th International Conference on Signal-Image Technology & Internet- Based Systems (SITIS), pp. 224 230, 2022.
Ahmet Iscen, Mathilde Caron, A. Fathi, and C. Schmid. Retrieval-enhanced contrastive vision-text models. ArXiv, abs/2300.071199 2023.
Aman Jain, Mayank Kothyari, Vishwaject Kumar, P. Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. Select, substitute, search: A new benchmark for knowledge-augmenneed visual question answering. Proceedings of the 44th International ACM SIGIR Conference on Research and Development Information Retrieval, 2021.
A. Jamshed and M. Fraz. Nlp meets vision for visual interpretation a retrospective insight and future dircc- tions. In 2021 International Conference on Digital Futures and Transformative Technologies (ICoDT2), pp. 1-8, 2021.
30

---

Published in Transactions on Machine Learning Research (01/2026)
Manas Jhalani, Annervaz K M, and Pushpak Bhattacharyya. Precision empowers, excess distracts: Visual question answering with dynamically infused knowledge in language models. ar preprint, MtXiv:2406.09994vv,, 2024.
Zhiwei Jia, P. Narayana, Arjun Reddy Akula, G. Pruthi, Haoran Su, Sugato Basu, and Varun Jampani. Kafa: Rethinking image ad understanding with knowledge-angmented feature adaptation of vision-language mod- els. ArXiu, abs/2305.18823 18373, 2023.
Chen Jiang, Masood Dehghan, and Martin Jägersand. Understanding contexts inside robot and human manipulation tasks through vision-language model and ontology system in video streams. 2020 TEEE/RS) International Conference on Intelligent Robots and Systems (TROS), pp. 8366-8372 2020.
Lei Jiang and Zuqiang Meng. Knowledge-based visual question answering using multi- modal semantic graph. Electronics, 2023.
Ruihan Jin, Ruibo Fu, Zhengqi Wen, Shuai Zhang, Yukun Liu, and Jianhua Tao. Fake news detection and manipulation reasoning via large vision-language models. ArXiv, abs/2407.0200 2024.
Liqiang Jing, Xuemeng Song, Kun Ouyang, Mengzhao Jia, and Liqiang Nic. Multi-source semantic graph- based multimodal sarcasm explanation generation. pp. 11849-11361. 2023.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. arXiv preprint, Sit.::1611.0689000 2016.
Penny Johnston, Keiller Nogueira, and Kevin Swingler. Ns-il: Neuro-symbolic visual question answering using incrementally learnt, independent probabilistic models for small sample sizes. IEEE Access, 11: 141406-141420, 2023.
Pankaj Joshi, Aditya Gupta, Pankaj Kumar, and Manas Sisodia. Robust multi model rag pipeline for documents containing lext, table & images. In 2024 3rd International Conference on Applied Arlificial Intelligence and Computing (ICAAIC), pp. 993 999,99999 2024.
Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models halluci- nate, 2025. CRL
04664
Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15624 15634, 2023.
Xuan Kan, Hejic Cui, and Carl Yang. Zero-shot scene graph relation prediction through commonsense knowledge integration. pp. 466-482, 2021.
Mahmoud Khademi,, Ziyi Yang, F. Frujeri, and Chenguang Zhu. Mm-reasoner: A multi-modal knowledge- aware framework for knowledge-basee based visual question answering. pp. 6571-658) 2023.
Tarun Khajuria, Braian Olmiro Dias, and Jaan Aru. How structured are the representations in transformer- based vision encoders? an analysis of multi-object representations vision-language models. ArXiu, 2906/2000.9967,, 2024.
M. A. Khaliq. P. Chang, M. Ma, B. Pffugfelder, and F. Miletí'c. Ragar, your falschood radar: Rag-augmented reasoning for political fact-checking using multimodal large language models. ArXiv, abs/2404. 12063, 2024.
M. J. Khan, Filip llievski, John G. Breslin, and Edward Curry. A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge. Neurosymbolic Artificial Intelligence, 2024.
Muhammad Jaleed Khan, J. Breslin, and E. Curry. Expressive scene graph generation using commonsense knowledge infusion for visual understanding and reasoning. pp- 93 112, 2022a.
31

---

Published in Transactions on Machine Learning Research (01/2026)
Muhammad Jaleed Khan, J. Breslin, and E. Curry. Neusire: Neural-symbolic image representation and enrichment for visual understanding and reasoning. 2022b.
Apoorv Khandelwal, Ellie Pavlick, and Chen Sun. Analyzing modular approaches for visual question de- composition. arXiv preprint, XrX:::23 .06411v1, 2023.
A. Kovalev, M. Shaban, Evgeny Osipov, and A. Panov. Vector semiotic model for visual question answering. Cognitive Systems Research, 71:52 63, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yan- nis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connect- ing language and vision using crowdsourced dense image annotations. arXiv preprinl, arXiv:1602.03332> 2016.
Jaeyun Lee and Incheol Kim. Vision- langunge geggungg-kowwldgee co-embedding for visual commonsense reasoning. Sensors (Basel, Switzerland), 21, 2021.
Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. Multimodal reasoning with multimodal knowledge graph. arXiv preprint, XXiv:2406.02030022 2024.
Paul Lerner, Olivier Ferret, C. Guinandcau, H. Borgne, Romaric Besançon, José G. Moreno, and Jesús Lovón- Melgarejo. ViQuAE, a Dataset for Knowledge-busel Visual Question Answering about Named Entities. 2022.
Paul Lerner, O. Ferret, and C. Guinandeau. Multimodal inverse cloze task for showledge-based visual question answering. pp. 569-587, 587, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021. URL https: org/ abs//bbb/200 11401
Bojin Li, Yan Sun, Xue Chen, and Xiangfeng Luo. Hkfnet: Fino-grained external knowledge fusion for fact-based visual question answering. 2024 International Joint Conference on Newral Networks (IJCNN), pp. 1-8, 2024a.
Guohao Li, Hang Su, and Wenwu Zhu. Incorporating external knowledge answer open-domain visual questions with dynamic memory networks. ArXiv, abs/1712.007333, 2017.
Guohao Li, Xin Wang, and Wenwu Zhu. Boosting visual question answering with context-awaro knowledge aggregation. Proceedings of the 28th ACM International Conference on Mullimedlia, 2020.
Hui Li, Peng Wang, Chunhua Shen, and Anton van den Hengel. Visual question answering as reading comprehension. arXio preprint, arXiv:1811.119038 2018.
Jiangmeng Li, Wenyi Mo, Wenwen Qiang, Bing Su, and Changwen Zheng. Supporting vision-language model inference with causality-pruning knowledge prompt. ArXiv, abs/2205.11100 2022a.
Meng Li, Tianbao Wang, Jiahe Xu, Kairong Han, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang, Shiliang Pu, and Fei Wu. Multi-modal action chain abductive reasoning. pp. 4617-4628, 2023a.
Mingxiao Li and Marie-Franne Francine Moens. Dynamic key-value memory enhanced multi-step graph reasoning for knowledgo-basee visual question answering. pp. 10983-10992, 2022.
Qifeng Li, Xinyi Tang, and Yi Jian. Learning to reason on tree structures for knowledge--bsee visual question answering, Sensors (Basel, Switzerland), 22, 2022b.
Qun Li, Fu Xiao, Le An, Xianzhong Long, and Xiaochuan Sun. Semantic concept network and deep walk- based visual question answering. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15:1 19, 2019.
32

---

Published in Transactions on Machine Learning Research (01/2026)
Qun Li, Fu Xiao, B. Bhanu, Biyun Sheng, and Richang Hong. Inner knowledge-basee img2doc scheme for visual question answering. ACM Transactions on Mullimedia Compuling, Communications, and Applicu- tions (TOMM), 18:1 21, 2022c.
Xin Li, Yu Zhang, Weilin Yuan, and Junren Luo. Incorporating external knowledge reasoning for vision- and-language navigation with assistant's help. Applied Sciences, 2022d.
Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. ArXiv, abs/2309.13625, 2023b.
Yili Li, Jing Yu, Keke Gai, and Gang Xiong. Tiu: Independent inference units for knowiedge-based visual question answering. pp. 109 120, 2024b.
Yumxin Li, Baotian Hu, Yunxin Ding, Lin Ma, and M. Zhang. A neural divide-and-conquer reasoning framework for image retrieval from linguistically complex text. Arviv, abs/2305.02222 02265, 2023c.
Zhuowan Li, Elias Stengel-Eskin, Yixiao Zhang, Cihang Xie, Quan Tran, Benjamin Van Durme, and Alan Yuille. Calibrating concepts and operations: Towards symbolic reasoning on real images. arXiv preprint, arXiv:2110.00519v/ 2021.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan Yuille. Super-clevr: A virtual benchnark to diagnose domain robustness in visual reasoning. arx preprint, S.Xiv:2212.002599: 2022e.
Ziyang Li, Jiani Huang, and Mayur Naik. Scallop: A language for neurosymbolic programming. arXiv preprint, arXiv:2304.0481211, 2023d.
Weixin Liang, Fei Niu, Aishwarya N. Reganti, G. Thattai, and Gökhan Tür. Lrta: A transparent neural-symbolic reasoning framework with modular supervision for visual question answering. ArXin, abs/2011.10731, 2020.
Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xiubo Geng, Yining Chen, Fan Liang, and Daxin Jiang. Maria: A visual experience powered conversational agent. orXiv preprint, arXiv:2105.1307322 2021.
Bingqian Lin, Zicong Chen, Mingjic Li, Haokun Lin, Hang Xu, Yi Zhu, Jian zhuo Liu, Wenjia Cai, Lei Yang, Shen Zhao, Chenfei Wu, Ling Chen, Xiaojun Chang, Yi Yang, L. Xing, and Xiaodan Liang. Towards med- ical artificial general intelligence via knovledge-enhanced multimodal pretraining. ArXiv, abs/2304.14204, 2023a.
Weizhe Lin and Bill Byrne. Retrioval augmented visual question answering with outside knowledge. arXiv preprint, AXX::22100008099 2022.
Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. ArXiv, Abs/2309.171333, 2023b.
Weizhe Lin, Zhilin Wang, and B. Byrne. Frqa 2.0: Introducing adversarial samples into fact-based visual question answering. ArXie, abs/ 0ts/2333.106999 2023c.
Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. Revive: Regional visual representation matters in knowledge-based visual question answering. ArXiu, abs/2206.01201, 2022.
Jiawei Liu, Jingyi Xie, Fanrui Zhang, Qiang Zhang, and Zhengjun Zha. Knowledge-enhanced hierarchical information correlation learning for multi-modal rumor detection. ArXiu, abs/2306.1590 2023a.
Luping Liu, Meiling Wang, Xiaohai He, L. Qing, and Honggang Chen. Fact-based visual question answering via dual-process system. Knowl. Based Syst., 237:107650, 2021.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xucyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents. Xie preprint, STT.:2311.05437v 2023b.
33

---

Published in Transactions on Machine Learning Research (01/2026)
Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Rar: Retrieving and ranking augmented milms for visual recognition. ar preprind, arXiv:2403.13805w 2024.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-We Chang, Y. Wu, Song- Chun Zhu, and Jianfeng Gao.
Chameleon: Plug-and-play compositional reasoning with large language models. ArXiu, abs/ hbss/230409984 2023.
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised visual-rettriever-eedde for knowledge-based question answering. pp. 6417-6431, 2021.
Maria Lymperaiou and G. Stamou. A survey on knowledge-enhanced multimodal learning. Artif. Intell. Rev., 57:284, 2022.
Xuan Ma, Xiaoshan Yang, and Changsheng Xu. Multi-source knowledge reasoning graph network for multi- modal commonsense inference. ACM Transactions on Multimedia Computing, Communications and Ap- plications, 19:1 - 17, 2022.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. orXiv preprint, arXiv:1906.0006722 2019.
Kenneth Marino, Xinlei Chen, Devi Parikh, A. Gupta, and Marcus Rohrbach. Krisp: Intograting implicit and symbolic knowledge for open-domain knowledge-based vqa. 2021 IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 14106 14106-14116, 2020.
Thomas Mensink, J. Uijlings, Tluís Castrejón, A. Goel, Felipe Cadar, Howard Zhou, Fei Sha, A. Araújo, and V. Ferrari. Encyclopedic vqa: Visual questions about detailed propertics of fine-grained categorics. 2023 IEEE/CVP International Conference Computer Vision (ICCV), pp. 3090 3101, 2023.
Guang ming Xian, Wencong Zhang, Fucai Lan, Yifan Lin, and Yanhang Lin. Multimodal knowledge triple ex- traction based on representation learning. In 2023 5th International Conference on Flectronic Engineering and Informatics (EET), pp. 684-689, 2023.
Aakansha Mishra, S. S. Miriyala, and V. N. Rajendiran. Learning representations from explainable and con- nectionist approaches for visual question answering. ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6420 6424, 2024.
Tautvydas Misiunas, Hassan Mansoor, Jasper Uijlings, Oriana Riva, and Victor Carbune. Vqa training sets are self-play environments for generating few-shot pools. ArXiv, abs/2405.19773, 2024.
Ludovico Mitchener, David Tuckey, Matthew Crosby, and A. Russo. Detect, understand, act neuro-symbolic hicrarchical reinforcement learning framework. 2021.
Aditya Mogadala. Multi-view representation learning for unifying languages, knowledge and vision. 2019.
Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thouue reasoning. pp. 18798-18806, 2024.
Guglielmo Montone, J. O' O'Regan, and A. Terekhov. Hyper-dimennional computing for a visual question- answering system that is trainable end-to-end. ArXiv, abs/1711.10185, 2017.
A. Mostafa, Hazem M. Abbas, and M. Khalil. Comparative study of visual question answering algorithms. In International Conference on Communication and Electronics Systems, pp. 1-6, 2020.
Medhini Narasimhan and Alexander G. Schwing, Straight to the facts: Learning knowledge base retrieval for factual visual question answering. arXiv preprint, arXiv:1809.01122v1, 2018.
Medhini Narasimhan, Svetlana Lazebnik, and Alexander G. Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. ar preprint, arXiv:1811.00555 1003388,, 2018.
34

---

Published in Transactions on Machine Learning Research (01/2026)
Abhishek Narayanan, Abijna Rao, Abhishek Prasad, and N. S. Vqa as factoid question answering problem: A novel approach for knowledge-aware and explainable visual question answering. Image Vis. Comput., 116:104328, 2021.
Sanika Natu, Shounak Sural, and Sulagna Sarkar. External commonsense knowledge as a modality for social intelligence questio--anseeing. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 3036 3042, 2023.
Zhe Ni, Xiao-Xin Deng, Cong Tai, Xin- Yue Zhu, Xiang Wu, Y. Liu, and Long Zeng. Grid: Sceno-graph-basse basod instruction-driien robotic task planning. ArXio, abs/2309.07726, 2023.
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. ArXiv, abs/2402.0794/ 07944, 2024.
Tomohiro Ogawa, Kango Yoshioka, Ken Fukuda, and Takeshi Morita. Prediction of actions and places by the time series recognition from images with multimodal 1lm. 2024 IEEE 18th International Conference on Semantic Compuling (ICSC), pp. 294 300, 2024.
Kun Ouyang, Yi Liu, Shicheng Li, Ruihan Bao, Keiko Harimoto, and Xu Sun. Modal-adaptive knowledge- enhanced graph-based financial prediction from monetary policy conference calls with Ilm. ArXiv, abs/2403.1 16055, 2024.
Trilok Padhi, Ugur Kursuncu, Yaman Kumar, V. Shalin, and Lane Peterson Fronczek. Improving contextual congruence across modalities for effective multimodal marketing using knowlodge-infused learning. ArXiu, abs/ abs/2442.03307, 2024.
Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamsoer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, Roger Chou, Julie Glanville, Jeremy M Grimshaw, Asbjørn Hróbjartsson, Manoj M Lalu, Tianjing Li, Elizabeth W Loder, Evan Mayo- Wilson, Steve McDonald, Luke A McGuinness, Lesley A Stewart, James Thomas, Andrea. C Tricco, Vivian A Welch, Penny Whiting, and David Moher. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372m711, March 2021. ISSN 0959-8138, 1833. doi: 10.1136/ bmj.n71. URL https://doi /tttp:://do..org10... 111.:/..13//mjm..nt........ n71 Epub 2021-03-29.
A. Potapov, A. Belikov, V. Bogdanov, and Alexander Scherbatiy. Cognitive module networks for grounded reasoning. pp. 148 158, 2019.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URL https://arxiv.. org/ abs/ 2207 16789
Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul A. Crook, Y. Xu, Xin Luna Dong, Christos Faloutsos, Lei Li, Babak Damavandi, and Scungwhan Moon. Snapntell: Enhancing entity-centric visual question answering with retrieval augmented multinodal Ilm. ArXiv, has/2403.04733, 2024.
Xiaoye Qu, Qiyuan Chen, Wci Wei, Jiashuo Sun, and Jianfeng Dong. Alleviating hallucination in large lasion-language models with active retrieval augmentation. ArXie, abs/2408.0055 00555,,055, 2024.
Zhaowei Qu, Luhan Zhang, Xiaoru Wang, Bingyu Cao, Yueli Li, and Fu Li. Ksf-st: Video captioning based on key semantic frames extraction and spatio-tempora, attention mechanism. 2020 International Wireless Communications and Mobile Computing (IWCMC), pp. 1388-1393 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL http::htt///aae http//xxivv.... wttpri/.rrr/bb//21...00
Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, and Tanuja Ganu. Retrieval augmented chest x-ray report generation using openai gpt models. arXiv preprint, arXiv:2305.03600v1 2023.
35

---

Published in Transactions on Machine Learning Research (01/2026)
Jiahua Rao, Zifei Shan, Long Liu, Yao Zhou, and Yuedong Yang. Retrieval-based knowledge augmented vision language pre-training. Proceedings of the 31st ACM International Conference on Mullimedia,, 2023.
Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. Vle-bert: Visual question 2022.
answering with contextualized commonsense knowledge. arXiv preprint,
Benjamin Z. Reichman, Anirudh S. Sundar, Christopher Richardson, Tamara Zubatiy, Prithwijit Chowdhury, Aaryan Shah, Jack Truxal, Micah Grimes, Dristi Shah, Woo Ju Chee, Saif Punjwani, Atishay Jain, and Larry Heck. Outside knowledge visual question answering version 2.0. ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5, 2023.
Heather Riley and M. Sridharan. Integrating non-monolonic logical reasoning and inductive learning with deep learning for explainable visual question answering. Frontiers in Robotics and AI, 6, 2019.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong, Interpretable machine learning: Fundamental principles and 10 grand challenges, 2021. URL httpps:://///xii http:/awwvvvoo.o/aaae.ee abs/
2103. 11251
Mrinmaya Sachan. Towards Literate Artificial Intellignnce. Phd thesis, Carnegic Mellon University, 2020. URL attps://doi. wtt//....14/////11.... 11144/11/11999377..
Alireza Salemi, Juan Altmaaye Pizzorno, and Hamed Zamani. A symmetric dual encoding dense retrieval framework for knowledge-intensive visual question answering. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023a.
Alireza Salemi, Mahta Rafiee, and Hamed Zamani. Pre-Trate Training Multi-Modaa Dense Retrievers for Outside- Knowledge Visual Question Answering. 2023b.
Sergio Sánchez Santiesteban, Sara Atito, Muhammad Awais, Yi-Zhe Song, and Josef Kittler. Improved image captioning via knowledge graph-augmented models. ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4290-429-0 4294, 2024.
Racid Saqur and Karthik Narasimhan. Multimodal graph networks for compositional generalization in visual question answering. 33, 2020.
Timo Schick, Jane Dwivedi-Vis Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL http://pe:///rd...o///////////// 04761
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv preprinl, arXiv:2206.01718x.] 2022.
Henry Senior, G. Slabaugh, Shanxin Yuan, and L. Rossi. Graph neural networks in vision-language image understanding: survey. ArXio, abs/ Abs/2303.03761, 2023.
Muralikrishnna G. Sethuraman, Ali Payani, Faramarz Fekri, and J. Clayton Kerce. Visual question answering based on formal logic. arXiv preprint, arXiv:2111.04785v1. 2021.
Sanket Shah, Anand Mishra, N. Yadati, and P. Talukdar. Kvqa: Knowledgo-aware visual question answering. pp. 8876 8884, 2019.
Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, and Jimmy Lin. Unirag: Universal retrieval augmentation for multi-modal large language models. ArXiv, abs/ 0////240..11111 2024.
Xiangqing Shen, Yurun Song, Siwei Wu, and Rui Xia. Ved: Knowledge base guided visual commonsense discovery in images. arXiv preprint, arXiv:2402.12213v11, 2024.
Violetta Shevchenko, Damien Teney, Anthony Dick, and Anton van den Hengel. Reasoning over vision and language: Exploring the benefits of supplemental knowledge. arXiv preprint, TTt::2101000013: 2021.
36

---

Published in Transactions on Machine Learning Research (01/2026)
Weimin Shi, Denghong Gao, Yuan Xiong, and Zhong Zhou. Qr-clip: Introducing explicit knowledge for location and time reasoning. ACM Transactions on Mullimedia Computing, Communications, and Appli- cations, 2024.
Zhan Shi, Yilin Shen, Hongxia Jin, and Xiao-Dan Zhu. Improving zero-shot phrase grounding via reasoning external knowledge and spatial relations. pp. 2253-2261, 2022.
Keisuke Shirai, C. C. Geltran-Hernandeee Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. Vision-languaal interpreter for robot task planning. 2024 IEEE International Conference on Robolics and Aulomation (ICRA), pp- 2051-2058, 2023.
A. Singh, Anand Mishra, Shashank Shekhar, and Anirban Chakraborty, From strings to things: Knowledge- enabled vqa model that can read and reason. 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 4601 4611, 2019.
Gursimran Singh. A bayesian approach to visual question answering. 2018.
Lingyum Song, Jianao Li, J. Liu, Yang Yang, Xuequn Shang, and Mingxuan Sun. Answering knowledge-based visual questions via the exploration of question purpose. Pattern Recognit., 133:109015, 2022a.
Zijie Song, Zhenzhen Hu, and Richang Hong. Efficient and self-adaptive rationale knowledge base for visual commonsense reasoning. Multimedia Systems, 29:3017-3033 3026, 2022b.
Zijie Song, Wenbo Hu, Hao Ye, and Richang Hong. How to use language expert to assist inference for visual commonsense reasoning. 2023 IEFE International Conference on Data Mining Workshops (ICDMW), pp- 521 527, 2023.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge, 2018. URL https: wwwg/a///////// 03975
Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting. Neural concept binder. arXiv preprint, M.KKv:2406.09944999 2024.
Gabricla Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohckar, Anahita Bhiwandiwalla, Shao- Yen Tseng, Matthew Lyle Olson, Yaniv Gurwiez, Chenfei Wu, Nan Duan, and Vasudev Lal. Twlm-interpret: An interpretability tool for large vision-language models, 2024. URL
03118
Xin Su, Man Luo, Kris W Pan, Tien Pei Chou, Vasudev Lal, and Phillip Howard. Sk-vqa: Synthetic knowledge gencration at scale for training context-angmented multimodal 1lms. ArXiv, 0bs//2406119588 2024.
Zhou Su, Chen Zhu, Yinpeng Dong, Dongqi Cai, Yurong Chen, and Jianguo Li. Learning visual knowledge memory networks for visual question answering. arXiv preprint, arXiv:1806.00 00066000.. 2018.
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. Modular visual question answering via code genera- tion. arXie preprint, arXiv:2306.0533 053322,,, 2023.
J. Sung, Qiuyuan Huang, Yonatan Bisk, Subhojit Som, Ali Farhadi, Yejin Choi, and Jianfeng Gao. Ink Indensive-neurrl--nwweeed aligned image text retrieval. 2022.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint, arXiv:2308.081228 2023.
Sinan Tan, Mengmeng Ge, Di Guo, Huaping Liu, and F. Sun. Knowledge-based embodied question answering, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:1 15:11948-111 11960, 2021.
P. Torino, Elena Baralis, and Dott. Andrea Pasini. Semantics-aware vqa sceno-graph-basee based approach enable commonsense reasoning. 2020.
37

---

Published in Transactions on Machine Learning Research (01/2026)
D. Tsatsou, Konstantinos Karageorgos, A. Dimou, J. Rubiera, J. M. López, and P. Daras. Towards unsu- pervised knowledge extraction. 2021.
Ben Zion Vatashsky and Shimon Ullman. Understand, compose and respond answering visual questions by composition of abstract procedures. arXiv preprint, arXiv:1810.106560 2018.
Ramakrishna Vedantam, Karan Desai, Stefan Lee, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Prob- abilistic neural-symbolic models for interpretable visual question answering. pp. 6428 6437, 2019.
Lara Verheyen, Jérôme Botoko Ekila, Jens Nevens, Paul Van Eecke, and Katrien Beuls. Neuro-symbolic procedural semantics for reasoning-intensiig visual dialogue tasks. pp. 2419-2426, 2023.
P. Vickers, Nikolaos Aletras, Emilio Monti, and Loic Barrault. In factuality: Efficient integration of relevant facts for visual question answering. pp. 468-475, 2021.
Duc Minh Vo, Hong Chen, Akihiro Sugimoto, and Hideki Nakayama. Noc-rek: Novel object captioning with retrieved vocabulary from external knowledge. arXiv preprinl, arXiv=2203.1449990 2022.
Mohammad Saif Wajid, Hugo Terashima-Marin, Peyman Najafirad, and M. A. Wajid. Deep learning and knowledge graph for image/vides captioning: A review of datasets, evaluation metrics, and methods. Engineering Reports, 6, 2023.
Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, J. B. Tenenbaum, and Chuang Gan. Sok-bench: A situated video reasoning benchmark with aligned open- world knowledge. 2024 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 13384-13394 2024a.
Bin Wang, Fuyong Xu, Peiyu Liu, and Zhenfang Zhu. Hypermr: Hyperbolic hypergraph multi-hop reasoning for knowledge-based visual question answering. pp. 3505-851,, 2024b.
Jianfeng Wang, Anda Zhang, Huifang Du, Haofen Wang, and Wenqiang Zhang. Knowledge-Pnthaneed Enhanced Visual Question Answering with Molti-modal Joint Guidance. 2022a.
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture
worth a thousand words? delving into spatial reasoning for vision language models, 2024c. URL https:
//arxiv.org/atee otgg/bs///0/...... 14852
Peng Wang, Qi Wu, Chunhua Shen, A. Dick, and A. Hengel. Explicit knowledge-band based reasoning for visual question answering, ArXie, abs/15 hts/1511.02570, 2015.
Peng Wang, Qi Wu, Chunhua Shen, A. Dick, and A. van den Hengel. Frqa: Fact-based visual question answering. IEEE Transactions Pattern Analysis and Machine Intelligence, 40:2413 2427, 2016.
Weizhi Wang, Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Visually-angmeented language modcling. ArXiu, abs/2205.10178, 2022b.
Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Composi- tional 4d dynamic scenes understanding with physics priors for vidco question answering. arXiv preprint, M:Xiv:240600062211 2024d.
Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, and Jure Leskovec. Vqa-gan: Reasoning with multimodal knowledge via graph neural networks for visual question answering. arXiv preprint, arXiv:2205.115014 2022c.
Zhu Wang, Sourav Medya, and Sathya N. Ravi. Differentiable outlier detection enable robust deep multi- modal analysis. Xiv preprint, arXiv:2302.056088/ 2023.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thouhht prompting elicits reasoning in large language models, 2023. URL http:://axxivorrrraa////11111110 https://arxiv http://a////00..1000000
38

---

Published in Transactions on Machine Learning Research (01/2026)
Jiahui Wei, Zhixin Li, Jianwei Zhu, and Huifang Ma. Enhance understanding and reasoning ability for image captioning. Applied Intelligence, 53:2706- 27222 2022.
Haoyang Wen, Honglei Zhuang, Hamed Zamani, Alexander mauplmann, and Michael Bendersky. Multimodal reranking for snowledge-intensive visual question answering. ArXiv, abs/2407.12277. 2024.
Zhang Wen and Yuxin Peng. Multi-level knowledge injecting for visual commonsense reasoning, IEEE Transactions on Circuits and Systems for Video Technology, 31:1042-1054, 2021.
Weixi Weng, Jieming Zhu, Hao Zhang, Xiaojun Meng, Rui Zhang, and Chun Yuan. Learning to compress contexts for efficient knowledge- based visual question answering. 2024.
Mingyuan Wu, Jingchong Yang, Jize Jiang, Mcitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengx- iang Zhai, and Klara Nahrstedt. Vtool-r1: Vims learn to think with images via reinforcement learning on multimodal tool use, 2025, URL htttp::/ wwexxii..rrg/eeo aabb/25055 19255
Qi Wu, Chunhua Shen, Anton van den Hengel, Peng Wang, and Anthony Dick. Image captioning and visual question answering based on attributes and external knowledge. arXiv preprint, S.XXv::603.02814420 2016.
Sen Wu, Guoshuai Zhao, and Xueming Qian. Resolving zero-shot and fact-based visual question answering via enhanced fact retrieval. IEEE Transactions Multimedia, 26:1790 1800, 2024a.
Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, V. loannidis, Karthik Subbian, J. Leskovec, and James Zou. Avalar: Optimizing Ilm agents for tool-assisted knowledge retrieval. ArXiv, hts//400.11200, 2024b.
Xiaoqian Wu, Yong-Lo Lu Li, Jianhua Sun, and Cewu Lu. Symbol-lim: Leverage language models for symbolic system in visual human activity reasoning. arXie preprint, WKIv:2311.17365V 2023.
Zeyu Xi, Ge Shi, Xuefen Li, Junchi Yan, Zun Li, Lifang Wu, Zilin Liu, and Liang Wang. Knowledge guided entity-aware video captioning and basketball benchmark. 2024.
Shouguan Xiao and Weiping Fu, Visual relationship detection with multimodal fusion and reasoning, Sensors (Basel, Swilzerland), 22, 2022.
Chunpu Xu, Min Yang, Chengming Li, Ying Shen, Xiang Ao, and Ruifeng Xu. Imagine, reason and write: Visual storytelling with graph knowledge and relational reasoning. pp. 2022-3029, 2021.
Jialiang Xu, Michael Moor, and Jure Leskovec. Reverse image retrieval cues parametric memory in multi- modal lmss. arXie preprint, arXiv:2405.1888 18744vv 2024a.
Ruinian Xu, Hongyi Chen, Yunzhi Lin, and Patricio A. Vela. Sgl: Symbolic goal learning in a hybrid,
modular framework for human instruction following. arXiv preprint, arXiv:2202.12912x.. 2022.
Wenjia Xu, Zijian Yu, Yixu Wang, Jiuniu Wang, and Mugen Peng. Rs-agent: Automating remote sensing tasks through intelligent agents. arXie preprint, arXiv:2406.07089v1 2024b.
Keyang Xuan, Li Yi, Fan Yang, Ruochen Wu, Y. Fung, and Heng Ji. Lemma: Towards Ivlm-enhanced multimoaal misinformation detection with external knowledge augmentation. ArXiv, abs/2402.119443, 2024.
Dizhan Xue, Shengsheng Qian, and Changsheng Xu. Integrating neural-symbolic reasoning with variational causal inference network for explanatory visual question answering. IEEE transactions on pottern analysis and machine intelligence, PP, 2024.
Yibin Yan and Weidi Xic. Echosight: Advancing visual-language models with wiki knowledge. ArXiu,
abs/2407..1733. 2024.
Jianwei Yang. Jiayuan Mao, Jiajun Wu, Devi Parikh, David Cox, J. Tenenbaum, and Chuang Gan. Object- centric diagnosis of visual reasoning. ArXio, abs/2012.11558 2020.
39

---

Published in Transactions on Machine Learning Research (01/2026)
Pengcheng Yang, Fuli Luo, Peng Chen, Lei Li, Zhiyi Yin, Xiaodong He, and Xu Sun. Knowledgeable storyteller: A commonsense-driven generative model for visual storytelling. pp. 5356-5362, 2019.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa, 2022. URL https://arxiv wwteaiiii.rrrgroeoo ht//abbs/100 05014
Zhenyu Yang, Lei Wu, Peian Wen, and Peng Chen. Visual question answering reasoning with external knowledge based on bimodal graph neural network. Electronic Research Archive, 2023.
Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doracmongpt: Toward under- standing dynamic scenes with large language models (exemplified as a video agent). or preprind, MxXI::2401.0639244 2024.
Keren Ye, Mingda Zhang, and Adriana Kovashka. Breaking shortcuts by masking for robust visual reasoning. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3519-3529, 2021.
Kexin Yi, Jiajun Wu, Chuang Gan, A. Torralba, Pushmeet Kohli, and J. Tenenbaum. Neural-symbolic vqa: Disentangling rcasoning from vision and language understanding. pp. 1039-1050, 2018.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprind, arXiv:191000144222, 2019.
Chengxiang Yin, Zhengping Che, Kun Wu, Zhiyuan Xu, and Jian Tang. Multi-clue reasoning with memory augmentation for knowledge-based visual question answering. ArXiv, abs/2312.127233, 2023.
D. Yu, Jianlong Fu, Xinmei Tian, and Tao Mei. Multi-source multi-level attention networks for visual question answering. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15:1 - 20, 2019.
J. Yu, Zihao Zhu, Yujing Wang, Weifeng Zhang, Yue Hu, and Jianlong Tan. Cross-modal knowledge reasoning for knowledge-basee based visual question answering. ArXiu, abs/2009.0012 00049, 2020.
Weijiang Yu, Haofan Wang, G. Li, Nong Xiao, and Bernard Ghanom. Knowledge-aware global reasoning for situation recognition. IEEE Transactions on Pallern Analysis and Machine Intelligence, 45:8621 -6633, 2023.
Lifan Yuan, Yangyi Chen, Xingvao Wang, Yi R. Fung, Hao Peng, and Heng Ji. Craft: Customizing ilms by creating and retrieving from specialized toolsets. arXiv preprint, arXiv:2309.1742855 2023a.
Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang, and Songfang Huang, Ramm: hetrieval-augmenteed biomedical visual question answering with multi-modal pre-training. Proceedings of the 31st ACM International Conference on Multimedia, 2023b.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual common-
sense reasoning. arXiv preprint, arXiv:1811.1080022 2018.
Huayi Zhan, Peixi Xiong, Xin Wang, Xin Wang, and Lan Yang. Visual question answering by pattern matching and reasoning. Neurocomputing, 467:323-333 2021.
Chunbai Zhang, Chao Wang, Yang Zhou, and Yan Peng. Vikser: Visual knowledge-driven self-reinfrrcing reasoning framework. XrXiv preprint, arXiv:2502.007110 2025a.
Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp. Can vision-language models be a good guesser? exploring vims for times and location reasoning. 2024 IEEE/CVF Winter Conference on Appli- cations of Computer Vision (WACV), pp. 625-634, 2023a.
Jiawei Zhang. Graph-toolformer: To empower lims with graph reasoning ability via prompt augmented by chatgpt. arXiv preprinl, arXiv:2304.111163,, 2023.
40

---

Published in Transactions on Machine Learning Research (01/2026)
Liyang Zhang, Shuaicheng Liu, Donghao Liu, Pengpeng Zeng, Xiangpeng Li, Jingkuan Song, and Lianli Gao. Rich visual knowledge-based augmentation network for visual question answering. JEEE Transactions Newral Networks and Learning Systems, 32:4362 4373, 2020.
Shunyu Zhang, X. Jiang, Zequn Yang, T. Wan, and Zengchang Qin. Reasoning with multi-structure com- monsense knowledge in visual dialog. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4599-4608, 2022a.
Xiaohan Zhang, Yan Ding, S. Amiri, Hao Yang, Andy Kaminski, Chad Esselink, and Shiqi Zhang. Grounding classical task planners via vision-language models. ArXiv, abs/2304.088877, 2023b.
Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Kuowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications, 14, 2023c.
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Peter Yu, Yuwei Bao, and Joyce Chai. Danli: Deliberative agent for following natural language instructions, arXiv preprint, Stti::220..24855V. 2022b.
Yichi Zhang, Jianing Yang, Jianing Yang, Keunwoo Peter Yu, Yinpei Dai, Jiayi Pan, N. Devraj, Ziqiao Ma, and J. Chai. Seagull: An embodied agent for instruction following through situated dialog. 2023d.
Yifeng Zhang, Ming Jiang, and Qi Zhao. Explicit knowledge incorporation for visual reasoning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1356-1365, 2021.
Yifeng Zhang, Ming Jiang, and Qi Zhao. Query and attention augmentation for knowledge-based explainable reasoning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15555- 15564, 2022c.
Yifeng Zhang, Shi Chen, and Qi Zhao. Toward multi- granularity decision-making Explicit visual reasoning with hierarchical knowledge. 2023 (EE//VV International Conference Computer Vision (ICCV), pp. 2573 2583, 2023e.
Yifeng Zhang, Ming Jiang, and Qi Zhao. Grace: Graph-basod contextual debiasing for fair visual question answering. In European Conference on Computer Vision, 2024. URL https://api. semanticscholar. org/CorpusID::724555
Yizhou Zhang, Loc Trinh, Defu Cao, Zijun Cui, and Y. Liu. Interpretable detection out-of-context misinformation with sirraa-symbollc-eehaneee large multimodal model. 2023f.
Zelan Zhang, Yi Ji, and Chunping Liu. Knowledge-aware causal inference network for visual dialog. Pro- ceedings of the 2023 ACM International Conference on Multimedia Retrieval, 2023g.
Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities, 2025b. URL //atpi://arxiv... wrg/ abaaabb///11 17385
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering lasion-language model with multi-modal in- context learning. ArXie, bbs/2309.07915, 2023a.
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. Retrieving multimodal information for augmented generation: A survey. arXio preprint, arXiv:2303. 10868v3, 2023b.
Yibiao Zhao. A quest for visual commonsense: Scene understanding by functional and physical reasoning. 2015.
Kaizhi Zheng, Jeshwanth Bhecmanpally, Bhrigu Garg, Scongsil Heo Dhananjay, Sonawane Winson, Chen Shree, Vignesh S Xin, and Eric Wang. Sage: A multimodal knowledge based conversational agent for complex task guidance. URL https://app. semanticacholar. hwtp:///....rroppppi.///26113113
41

---

Published in Transactions on Machine Learning Research (01/2026)
Wenfeng Zheng, Lirong Yin, Xiaobing Chen, Zhiyang Ma, Shan Liu, and Bo Yang. Knowledge base graph embedding module design for visual question answering model. Pattern Recognil., 120:108T 2021.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. arXiv preprint, WrXiv:2307.1386442 2023.
He Zhu, Ren Togo, Takahiro Ogawa, and M. Haseyama. Multimodal natural language explanation generation for visual question answering based on multiple reference data. Electronics, 2023.
Yi Zhu, Xiwen Liang, Bingqian Lin, Qixiang Ye, Jianbin Jiao, Liang Lin, and Xiaodan Liang, Configurable graph reasoning for visual relationship detection. IEEE Transaclions on Nearul Nelworks and Learning Systems, 33:117-129, 2020a.
Zihao Zhu. From shallow to deep: Compositional reasoning over graphs for visual question answering. arXiv preprint, arXiv:2206.15330 2022.
Zihao Zhu, J. Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu. Mucko: Multi- cross- moosl knowledge reasoning for fact-based visual question answering. ArXin, abs/2006.09073, 2020b.
M. Ziaeefard and F. Lécué. Towards gewwledge-augmentee visual question answering. pp. 1863-183, 2020.
42


Reviews:
[{"review_id": "jbmOK3SJlS", "reviewer": "Reviewer_xaWS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a survey article about vision-language models that are *augmented*, either with retrieval from a knowledge base or with symbolic computation from an external tool. Unusually for a ML survey, I think, it follows the PRISMA guidelines for systematic reviews of medical studies and gives a detailed record in the appendix of how the surveyed papers were collected. This methodology should also be considered one of the contributions of the paper.\n\nThe TMLR guidelines for surveys state that surveys should \"draw new connections, highlight trends, and suggest new problems.\" The categorization of previous approaches into late, middle, and early fusion, and into knowledge augmentation vs. symbolic computation is helpful and can be considered \"drawing new connections.\" The discussion in Section 6 can be considered \"highlighting of trends,\" and the future directions in Section 6.3 can be considered \"suggesting new problems.\"\n\nI feel that the survey is generally dense with citations and doesn't necessarily dig very deeply into them; for example, if there are design decisions that many of the surveyed approaches have in common, it would be worthwhile to present those in greater technical detail. In this respect, the value added by synthesizing all these citations may be low.\n\nSection 6.1 discusses five challenge areas. Unlike Sections 2-5, where the organizational principle is very clear, here I feel uncertain what the organizational principle is and whether this is an exhaustive list of challenge areas. It feels like there might be some principle(s) (e.g., space versus time) that are there, but not made explicit yet.", "claims_and_evidence_yn": "Yes", "claims_explanation": "As a survey article, it doesn't exactly have scientific claims, but it promises to systematically survey this area of research, and it does. As mentioned above, it goes to greater lengths than is usual in ML to convince the reader that the survey covers all papers in this area that were available at the time.", "audience_yn": "Yes", "audience_explanation": "As a survey article, it doesn't exactly have findings, but it surveys a topic squarely in the interest area of TMLR's audience. The categorization of these papers, and the discussion of present challenges and some solutions are also clearly in scope for TMLR.", "requested_changes": "The term \"augmented\" seems to me to be not very informative (indeed, at first glance, I thought this was a survey on general vision-language models). When I google \"augmented vision-language models,\" the first hit is this paper. Is there not a widely-used term for this area of research?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cg28EY5fHJ", "reviewer": "Reviewer_psmb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Vision-Language Models that take in text and image and output text have advanced rapidly in recent years.  Given that they are typically derived from LLMs, VLMs are also undergoing a trend where their shortcomings are addressed by augmenting their abilities with external systems, for example to perform precise math calculations or retrieve external knowledge.\n\nThis paper aims to fill a gap in the literature by surveying the state of the art in approaches to augmenting VLMs with external systems.  While previous surveys exist covering specific aspects of augmented VLMs (AVLMs), e.g., knowledge enhancement or the use of neuro-symbolic systems, this can stand as the first systematic study of the full landscape of AVLMs.\n\n## Strengths\n\nStrengths of this paper include:\n - Tool use in LLMs/VLMs is of very high interest to the community\n - The goal of performing the first systematic survey of the full landscape of AVLMs sets the stage for a meaningful contribution.\n\n## Weaknesses\n\nWhile I agree with the authors that this study has the potential to fill a meaningful gap in the literature, I find that the current version has significant flaws.\n\nFirst, I found the abstract and introduction to be very confusing given the contents of the paper.  They focus on neurosymbolic approaches to augmenting VLMs, but this is only one aspect of what’s covered in the paper.  For example, the enumeration of the weaknesses of neuro-symbolic approaches seems out of place, especially in the introduction.  The focus here, as I understand it, is instead addressing weaknesses of VLMs.  Such a detailed enumeration of the weaknesses of VLMs, ideally with visual examples, would have been useful early in the paper.  An illustrative Figure 1 would be useful (see below for more thoughts on figures).\n\nSecond, my biggest concern is that I find that many citations that I look up do not directly support the sentences or passages in which they are used.  This is a major concern for a survey.  Although I cannot practically review every citation in the paper, here are some examples that I found:\n\n - In the first sentence of the first paragraph, CLIP is cited to be considered as a VLM.  However, CLIP cannot perform the tasks being discussed, and does not meet the definition of a VLM given in Section 1.2, as it does not generate text.\n- Again in paragraph 1, Rudin 2021 and Mitchell 2022 are not strongly related to the points being made.  More specific references should be used, or these references should be positioned alongside statements more accurately reflecting their contents.  Mirzadeh et al 2024 is not about VLMs.  Overall, by my count only 2 of 6 citations from paragraph 1 are about VLMs.  Given the vast body of literature being produced on these models, it is unclear why stronger references cannot be found.\n - I looked at a set of citations from Section 4.2, and found that many did not strongly (or even weakly) support their accompanying passage.  For example, in the sentence about tool use: Hu et al., 2024 is not about  integrating external information or symbolic computation during the VLM’s forward pass, or interaction with the model’s intermediate representations; Fan et al., 2024 is not about tool use; Liu et al., 2023b does not seem to be about middle fusion; Hu et al., 2023c and Wu et al., 2024 are not about LLMs.\n - Suris 2023 does not involve a VLM.  I do not believe that any of its 4 references in the main paper are accurate vis-a-vis what is in this paper.\n - The first sentence of Section 7.6 has two references for AVLMs and tool use,  (Qin et al., 2023; Schick et al., 2023).  Neither of these references feature VLMs (or AVLMs).  I also believe that the Yang and Lu papers referenced in that section do not feature VLMs.  \n - Overall, I understand that LLM references are useful here, but the way they are used is very unclear, and usually makes it seem like the reference should be about multimodal models (at least way more often than they are.\n\nNext, I find that the paper lacks depth, such that the paragraphs become highly repetitive and not very informative.  Some problem setup sections (with accompanying visuals) to give basic technical background on VLMs, knowledge graphs, neuro-symbolic systems, tool use, etc. would be very helpful.  As it stands, while I am very familiar with LLMs/VLMs and tool use, after reading this survey I do not have a clear idea of how VLMs interact with KGs or symbolic systems.  More details on some select papers would be helpful in illustrating what is going on here.\n\nI am also confused about how many of these techniques are inference-only; for example, in Section 3.1 (bottom page 5) how does the model condition on retrieved encodings that it has never seen?  How do we incorporate an LSTM as an inference-time technique without training?\n\nI am further unsure of what constitutes an intermediate representation under the conception in this paper, and thus I have trouble understanding how approaches end up in Section 4.  Does a chain of thought constitute an intermediate representation?  Based on the methods included it seems so, but this does not match up with the description at the top of Section 4.  \n\nThis statement in 7.1 requires citation: “As of the time of writing, many commercially\navailable AI products use only early fusion retrieval techniques…”\n\nMore illustrative figures, especially high-level ones early in the paper, would be very beneficial.  Also, the amount of citations in some sentences make it hard to find the continuation of the sentence.  Some citations might be moved to tables, figures, or elsewhere.", "claims_and_evidence_yn": "No", "claims_explanation": "I find too many instances where citations do not support the claim that they are referenced in to support this paper’s acceptance in its current form.", "audience_yn": "Yes", "audience_explanation": "Yes, a systematic review of AVLMs would be of interest to the TMLR audience.", "requested_changes": "### Important Changes\n- Review all citations, make sure they more directly support the accompanying claims.\n- Rewrite abstract and introduction to clarify scope; better balance of information about VLMs and different augmentation approaches; less focus on shortcomings of neurosymbolic approaches.\n- Give some light technical background, and more depth in places in Sections 3-5, so that a VLM expert could gain a better understanding of how these techniques might work.  \n- Clarify the criteria for designation as middle vs. late stage fusion.\n\n### Other Suggestions\n- More illustrative figures.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NHZhbXXKEX", "reviewer": "Reviewer_CEXb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of scientific literature on augmented vision language models, i.e., vision-language models that make use of additional functionality beyond the VLM itself, such as prompt augmentation, retrieval, tool calls, or external solvers for generating answers. The paper categorizes augmentation into early-, middle-, and late-fusion and discusses related approaches. The paper concludes with a discussion on current challenges and possible future directions. The supplementary material documents in detail how literature to be included in the survey was collected, and also briefly comments on the limitations of the literature search.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper presents a survey of existing literature on augmented vision language models, but no empirical results. The methods to search and filter literature for inclusion are documented in the supplementary and follow good practices. The supplementary material also features tables organizing individual publications by keywords, which could serve as a useful resource for researchers seeking an effective reference guide for the field.", "audience_yn": "Yes", "audience_explanation": "The main use case of this paper is obtaining a comprehensive overview of the current state of augmented vision language models. This is certainly of interest to a part of TMLR's audience, as vision-language models represent a significant area of current AI/ML research.", "requested_changes": "**(R1, critical)** Sections use different citation styles. Section 1 and 7.5 use `\\citet`, while Sec. 3 and others `\\citep` . In most cases, `\\citep` is correct. Please thoroughly review references and adjust them to the context.\n\n**(R2, critical)** Sections 7.1 to 7.4 lack explanations that support the claims. For example, Sec. 7.2 claims \"hybrid architectures consistently outperform purely neural methods of similar computational requirements\". On which models/papers is this claim based? In Sec. 7.4, it is claimed that \"few systematically evaluate both the correctness and interpretability of neural-symbolic systems in a consistent manner\". Which are the few approaches that do this? This part is also a great opportunity to highlight promising directions towards unifying evaluation for augmented VLMs, beyond the insight that no generally accepted framework exists at this point? Overall, these sections would significantly benefit from adding explanatory statements and justifications.\n\n**(R3, critical)** Sec. 6 lists many datasets, most of which are not exclusive to augmented VLMs. Here, it is necessary to better highlight the connections of the identified dataset groups to augmented VLMs. How is their use for training or evaluating augmented VLMs different from their use for non-augmented VLMs?\n\n**(R4, critical)** In Sec. 7.7, are there any aspects that are specific to VLMs? Most proposed directions of future work appear to be valid as well for LLMs, for example, better integration with the environment, specialized applications, and human-in-the-loop scenarios. To better align this section with the audience, I recommend adding some specific directions targeting VLMs.\n\n**(R5, non-critical)** The main part of the survey is very concise, mostly explaining important keywords in 1 sentence and then citing relevant literature. I think it would be a really great addition to add a table to the supplementary material containing all keywords listed in Tables 1 to 4 and more detailed definitions of these keywords.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)