Imagination Augmented Generation: Learning to Imagine Richer Context for Question Answering over Large Language Models
Authors: Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, Shengping Liu, Jun Zhao


Reviewer paper summaries: This paper proposes Imagination-Augmented-Generation (IAG), a framework for knowledge augmentation utilizing LLMs' internal knowledge without relying on external resources.
This paper also proposes IMcQA, a QA method to better utilize LLMs' internal knowledge.
IMcQA consists of two modules: context generator and hypernetwork.
Context generator is trained to generate documents from questions using teacher-model's output documents, which are compressed by teacher models from retrieved external documents.
Hypernetwork is trained to generate LoRA adapter weights to be used to generate a better QA model. It also utilizes teacher model's (FiD) outputs as training signals.
FiD uses long context as inputs and hypernetwork is trained with short context, which I think would lower the computational cost at test time.
The authors argue that IMcQA addresses three issues of existing knowledge augmentation methods such as RAG and GAG: dependence on external resources, increased execution costs, and susceptibility to noise data at test time.
The authors evaluate two LLMs on three QA datasets in the experiments.
Summary of strengths: * The results show that the proposed method improves the scores by 2-3 points in IID and OOD settings while reducing computational costs at test time.
* The evaluation is conducted thoroughly.
Summary of weaknesses: * This paper is a little hard to read. From the Introduction and Method sections, it is unclear to me which part of the proposed method contributes to addressing the above three issues. E.g., which part of the method addresses susceptibility to noise data? The writing could be improved more.
* The proposed method may be too complex. E.g., the reason why hypernetwork should be used is unclear to me. The experimental results show the effectiveness of hypernetwork, but the motivation and justification to use hypernetwork seem lacking in the paper.
Comments, suggestions and typos: * Figure 3: The Architecture -> Figure 3: The architecture
* The difference between IAG and IMcQA is unclear to me.
Overall assessment scores: 2.5