TLDR: Token-Level Detective Reward Model for Large Vision Language
  Models
Authors: name: Deqing Fu, name: Tong Xiao, name: Rui Wang, name: Wang Zhu, name: Pengchuan Zhang, name: Guan Pang, name: Robin Jia, name: Lawrence Chen
Organization: Meta Llama

Although reward models have been successful in improving multimodal large
language models, the reward models themselves remain brutal and contain minimal
information. Notably, existing reward models only mimic human annotations by
assigning only one binary feedback to any text, no matter how long the text is.
In the realm of multimodal language models, where models are required to
process both images and texts, a naive reward model may learn implicit biases
toward texts and become less grounded in images. In this paper, we propose a
Token-Level Detective Reward Model
(TLDR) to provide fine-grained annotations to each text token. We
first introduce a perturbation-based method to generate synthetic hard
negatives and their token-level labels to train TLDR models. Then we show the
rich usefulness of TLDR models both in assisting off-the-shelf models to
self-correct their generations, and in serving as a hallucination evaluation
tool. Finally, we show that TLDR models can significantly speed up human
annotation by 3 times to acquire a broader range of high-quality vision
language data.