Test-time Vocabulary Adaptation for Language-driven Object Detection
Authors: Mingxuan Liu, Tyler L. Hayes, Massimiliano Mancini, Elisa Ricci, Riccardo Volpi, Gabriela Csurka


Reviewer paper summaries: Brief Summary: The paper tackles the task of open vocabulary object detection and proposes a test-time adaptation. The key idea is that open-vocabulary detectors have a large list of choices to choose from which can make it difficult to identity the appropriate object. To circumvent this issue, the authors propose using a separate Vision-Language Model to perform image captioning and apply a noun-detector over it and constrain the output object set. 

Experiments with different object detectors (Detic, VLDet, CoDet) on COCO and Object365 ovd datasets shows that the proposed test time adaptation (VocAda) is helpful and improves over competitive baselines including a strong RAM++ baseline which provides image tagging output often by margin of 1-3 AP50 points.
Summary of strengths: Pros: 

1. The paper tackles an important part in the task of open-vocabulary detection, namely, fixing the issue of test-time vocabulary set. As such, the paper is well motivated. 

2. The proposed method improves results compared to strong baselines such as RAM++ especially on AP50-novel. 

3. Authors provide detailed ablative study and visualizations comparing different variants of their method.
Summary of weaknesses: Cons: 

1. The comparison to RAM++ seems a bit unfair. It is essentially comparing the VLM to RAM++. A different choice of VLM would likely be worse than the RAM++ baseline, and it has little to do with other choices in VocAda method. 

2. The novelty is a bit limited. The idea is to mainly rely on the VLM to act as a strong image tagging baseline which acts as a constraint for the open-vocabulary detection model.

3. A big issue is the inference time is greatly increased due to processing via the VLM. The authors should provide some metrics on the extra-overhead time as well as computation cost. 

4. The authors should also compare on LVIS dataset, especially the rare category.

5. (Minor) The authors should compare with some other baselines such as Grounding-SAM [Ref1]. This can be used for comparison as well as a baseline for comparison. 

[Ref1]: Ren, Tianhe, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen et al. "Grounded sam: Assembling open-world models for diverse visual tasks." arXiv preprint arXiv:2401.14159 (2024).
Comments, suggestions and typos: The diagrams, tables, and visualizations are very impressively done. Kudos to the authors.
Overall assessment scores: 2.5