GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Authors: Shilong Zhang, Peize Sun, Shoufa Chen, Minn Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, Ping Luo


Reviewer paper summaries: The paper introduces a new instruction tuning method to enhance large language models (LLMs) by incorporating references to regions of interest (RoIs) in the instruction set. This aims for better region understanding by interleaving RoI features with text embeddings. The model, GPT4RoI, allows interaction through both language and visual inputs, such as drawing bounding boxes. The paper demonstrates high accuracy on benchmarks.
Summary of strengths: 1. The method combines language with RoI references, which seems to provide more accurate region-specific understanding.
2. The model shows good empirical performance with notable accuracy on several benchmarks.
3. The capability for interactions through both language and visual inputs could potentially enhance the user experience.
4. The release of code and data supports open science and further research.
Summary of weaknesses: 1. The evaluation of the model's generalist capabilities on a broader range of VQA benchmarks seems limited.
2. There appears to be a lack of detailed ablation studies, especially regarding the effectiveness of embedding RoI features.
Comments, suggestions and typos: 1. Including more experiments to showcase performance across various VQA benchmarks would strengthen the claims of generalist capabilities.
2. Detailed ablation studies on the RoI feature embedding could provide valuable insights.
Overall assessment scores: 2.5