Balancing Generalization and Specialization in Multimodal Retrieval and Multihop QA Systems
Authors: Peter Carragher, Raghav R, Abhinand Jha, Kathleen M. Carley


Reviewer paper summaries: This paper evaluates the performance of LLMs on multi-modal QA tasks, comparing general-purpose LLMs with specialised models. It utilises a multi-modal multi-hop QA architecture built upon existing models and assesses its performance against LLMs on the WebQA benchmark. The paper also explores the trade-off between zero-shot prompting and fine-tuning, finding that while LLMs excel in certain tasks, fine-tuned specialised models outperform them in closed domains and high-complexity tasks.
Summary of strengths: The paper addresses an important topic: LLMs performance on multi-modal QA tasks. It provides a comprehensive evaluation of LLMs and specialised models on the WebQA benchmark, offering insights into their strengths and weaknesses. The paper contributes to the multi-modal multi-hop QA field by effectively combining existing models. The analysis of retrieval and QA performance and task complexity provides insights into LLMs behaviour.
Summary of weaknesses: The paper lacks a comprehensive discussion of related work in multi-modal QA and LLMs, which would provide a broader context for its contributions. While it mentions models used in their methodology, a broader overview of related work is needed to fully situate their approach within the field. Additionally, the experiments are limited to the WebQA benchmark, and evaluating the proposed architecture on other multi-modal QA datasets would demonstrate its generalisability.
Comments, suggestions and typos: * Some descriptions are a bit too wordy (e.g., 095: eschewing, 100: disparate)
* 100: "In-Context" instead of "In-Content"
* 232-233: Authors mention the QA classifier, which was not introduced/discusses before.
* The paper's structure could be improved to enhance readability and make it easier for readers to follow the flow of information.
Overall assessment scores: 2.5