Mitigating Object Hallucination via Concentric Causal Attention
Authors: Yun Xing
Yiheng Li
Ivan Laptev
Shijian Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Object Hallucination
Multimodal Learning
Visual Hallucination
Primary area: machine_vision

Abstract: Recent Large Vision Language Models (LVLMs) present remarkable zero-shot conversational and reasoning capabilities given multimodal queries. Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs are prone to generate textual responses not factually aligned with image inputs. Our pilot study reveals that object hallucination is closely tied with Rotary Position Encoding (RoPE), a widely adopted positional dependency modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs tend to hallucinate more when relevant visual cues are distant from instruction tokens in the multimodal input sequence, Additionally, we observe a similar effect when reversing the sequential order of visual tokens during multimodal alignment. Our tests indicate that long-term decay in RoPE poses challenges to LVLMs while capturing visual-instruction interactions across long distances. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs by naturally reducing relative distance between visual and instruction tokens. With CCA, visual tokens can better interact with instruction tokens, thereby enhancing model's perception capability and alleviating object hallucination. Without bells and whistles, our positional alignment method surpasses existing hallucination mitigation strategies by large margins on multiple object hallucination benchmarks.

Mitigating Object Hallucination via Concentric Causal Attention
Yun Xing' Yiheng Li Ivan Laptev2
Shijian Lu' MBZUAI
Nanyang Technological University
Abstract
Recent Large Vision Language Models (LVLMs) present remarkable zero-shot conversational and reasoning capabilities given multimodal queries. Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs are prone generate textual responses not factually aligned with image inputs. Our pilot study reveals that object hallucination is closely tied with Rotary Position Encoding (RoPE), widely adopted positional dependency modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs tend to hallucinate more when relevant visual cues are distant from instruction tokens in the multimodal input sequence. Additionally, we observe similar effect when reversing the sequential order of visual tokens during multimodal alignment. Our tests indicate that long-tern terg-termmo decay in RoPE poses challenges to LVLMs while capturing visual- instruction interactions across long distances. We propose Concentric Causal Attention (CCA), simple yet effective positional alignment strategy that mitigates the impact of RoPE long-term decay in LVLMss by naturally reducing relative distance between visual and instruction tokens. With CCA, visual tokens can better interact with instruction tokens, thereby enhancing model's perception capability and alleviating object hallucination. Without bells and whistles, our positional alignment method surpasses existing hallucination mitigation strategies by large margins on multiple object hallucination benchmarks.
Introduction
Large Vision-Language Models (LVLMs) 46 45
have drawn increasing attention from the Al research community due their impressive power understanding the visual world and unprecedented ability to interact with humans via conversations. Their capability to process multimodal sequences has opened up new possibilities for a wide range of vision and language such as handling interleaved image-text inputs 4 35 and interactive user queries [82] However, existing LVLMs still suffer from object hallucination
15
tasks 32 2
14,, a tendency to generate inaccurate responses that not factually aligned with image inputs. Such phenomenon challenges the faithfulness and reliability of LVLMs in practical use, impeding their deployments to real-world applications [14].
wide range of approaches have been proposed mitigate object hallucination in LVLMs. One straightforward approach involves post-hoc correction using revisor models 77 33
reducing occurrences of hallucinated responses. Another viable approach improve supervised fine-tuning by diversifying instruction tuning data 43 additionally aligning model responses with human preference [62 76.. Despite their effectiveness in mitigating LVLM object hallucination, acquiring high-quality annotations can be labor-intensive, making these approaches costly to implement.
"Equal contribution
i Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(4] RuPE LVLMs
[b] S
Sestruge information flww RoPP
Sateaal S meraat information Row
Image
Instraction
Rolk
Vision kacoder
LLaMA Embeddings
Projection
Ralary Pusition Emooding
47
tokeal W Sezza Intraction bikes, W&S Startut
W/o long term decay
w/ long term decay
00
Figure 1: Long-term decay of RoPE [1] in Large Vision Language Models (LVLMS). (a) a schematic view of inference in LVLMs, typically involving pre-trained vision encoder, a large language model and a projector to map visual tokens to textual space. For each of V visual tokens Seision, we aggregate its information flow instruction tokens Sinstruct and reshape the aggregation results 2-D (vV by vV). Applying RoPE on visual tokens introduces long-term decay as illustrated in (c), referring to the phenomenon where information flowing from visual tokens to instruction tokens gradually decays from lower-right region (rightmost visual tokens in the 1-D sequence) to upper-left region (leftmost visual tokens). For instruction tokens, they have much less direct interaction with lefumost visual tokens as compared with rightmost visual tokens, leading to inferior multimodal alignment in the trained LVLMs. (b) and (c) are derived from the adversarial subset of the 3k POPE [4]] image-instruction pairs. Best viewed in color.
Recently, several studies explore training-free mitigation of object hallucination by rectifying fallacies in LVLM autoregressive decoding 26 34
However, the need compare among many candidates inevitably slows down the decoding process, making these approaches less efficient during inference.
Distinct from previous efforts, we attend to Rotary Position Encoding (RoPE) [1], widely used positional dependency modeling design in LVLMs 46 84), and investigate how may affect object
hallucination in LVLMs. Similar sinusoidal function [65], RoPE is proposed to encode position information into representations, enhancing model's 's ability in understanding sequential order of input tokens. In spite of its success modeling natural language [53] 63 [4], this design leads to long-term decay [61] in multimodal alignment, a phenomenon where information flow from visual tokens to instruction token gradually diminishes with increasing relative distance.
We analyze the impact of long-term decay [6] 53) on LVLMs. For every visual token in a multi- modal sequence, aggregate its information flow all instruction tokens and examine how these aggregations distribute across all visual tokens. As presented, in contrast to information flows of visual tokens without RoPE (Fig. TT (b)), applying RoPE attenuates information flows of leftmost visual tokens, which are located the farthest from instruction tokens in the sequence (Fig. | (c)). Such long-term&en decay benefits natural language modeling [61]. but induces insufficient interactions between visual tokens and instruction tokens, leading to inferior multimodal alignment and object hallucinations in the trained LVLMS (see our experiments in Sec. 3 for details).
To this end, we propose Concentric Causal Attention (CCA), a novel position alignment method for training LVLMs with mitigated object hallucination. CCA consists of a position reorganization module for visual tokens and an accompanying causal mask rectification module for modeling 2-D continuous positional dependency. Instead of following raster-scan sequential order of existing LVLMs, CCA starts from peripheral of 2-D images and ends centers. Such position alignment strat- egy enjoys two merits: 1) relative distance from instruction tokens to visual tokens significantly reduced, alleviating limitations brought by long-term decay in RoPE; 2) rectified causal attentions follow 2-D spatial locality of images, compared to 1-D causal attention originally designed for natural languages. We carry out pre-traee pre-training and instruction tuning as [46] and verify our trained model multiple object hallucination benchmarks [4] 57 20 (+4.24% Accuracy and +2.73% on F1 score, as compared to the state-of-the-ee method
34 on POPE). From a broader perspec-
I Information flow here refers to self-attentions from instruction tokens to visual tokens. 2--D image tokens flattened from left to right, top bottom, into 1-D visual token sequence.

---

tive, our method also improves general perception capability of LVLMs. Preliminary experiments show that our positional alignment approach surpasses the baseline consistently over 6 multimodal benchmarks 33
48
22
28
49
8
Our contributions are three-fold. First, we perform in-depth analysis correlation between rotary position encoding and object hallucination in large vision-language models. Second, motivated by our analysis, we propose Concentric Causal Attention (CCA), a simple yet effective method to mitigate LVLM object hallucination caused by RoPE long-term decay. Third, experiments on multiple benchmarks and comparisons with the state-of oftt--ofthh-arr methods support efficacy of our design.
2 Related Works
Large Vision Language Models. Language modeling has made notable progress in recent years, evolving from robust representation models [1
to instruction-tunee conversational chat- bots 663 64 12, M. These achievements have driven research in creating Large Vision Language
56
5.
Models (LVLMs) that can manage multimodal inputs 72 46
45
84 7
40
[ 39]. Pioneering
studies in this field (2 4 38 37 connect a vision-only encoder with a powerful frozen language-only
model to bridge modality gap, enabling dense interactions across visual and textual features. Powered by instruction-tuned LLMs [22 LLaVA [46,, InstructBLIP [15] and MiniGPT4 (84) allow interactive conversations between trained models and users. On top of these studies, LVLMs are empowered with more advanced capabilities. such as engaging in referential dialogues (7] 74 g 54 (77), handling
interleaved image-text data [2] 4 35] or understanding visual prompts, like point or box inputs from
users 54 77.. Despite advancements in LVLMs, many of these models still generate inaccurate responses not aligned with visual inputs.
Object Hallucination refers to common problem of existing LVLMs [14
68
19
66). Specifically, LVLMs tend to generate inaccurate responses that are not factually aligned with image inputs. To address this issue, several recent explorations 7
S 33 resort to post-hoc correction of model hallucinated outputs. These methods rely on either external models 47 to correct
hallucinated responses (73) or self-correction techniques [3 70 However, both of these methods
break end-to-end inference scheme. contrast, [43] 76 62, 29 78 75) ground their approaches on improving instruction tuning, by either diversifying instruction data or aligning model responses with human feedback. However, acquisition of more instruction data or preference data is labor-intensive. Recently, several studies attempt to mitigate object hallucination in training-fire frre manner [26 34 [0].
However, the need to compare among many candidates inevitably slows down the decoding process, making these approaches less efficient during inference. From a distinct perspective, we ground our design in correlation between widely adopted rotary position encoding and object hallucination.
Position Encoding in Transformers. Transformer models (65) do not inherently comprehend sequen- tial information of imput tokens, which is inferior for modeling sequential data like natural language as compared to recurrent structures like (24). To mitigate this issue. [65] introduces sinusoidal position encodings to incorporate position information to input embeddings. In addition, several studies resort to learnalle position encodings 18], which allow their models to update positional parameters during training. In contrast to absolute position encodings, relative position encodings on relative position among tokens. They integrate position information in self-attentions, presenting potential for modeling sequences with variable lengths 53]. Among these studies, Rotary Position Encoding (RoPE) [61] encodes position information by multiplying input embeddings with rotation matrices. In comparison to other position encoding designs, RoPE is capable of equipping linear self-attention with relative position encoding, which proven effective for pre-training large language models (63] 64). A few recent studies explores RoPE for vision tasks [13] 50 69] showcasing its potential to domains beyond natural language. In this paper, we investigate the role of RoPE in LVLMs and how it affects object hallucination in these models.
(59 31 23 27 focus
Motivation
In this section, we further examine the long-term decay in RoPE and conduct quantitative analyses to illustrate its correlation with object hallucination. We begin with brief introduction to the widely adopted LVLM architecture and how RoPE (1]] is applied in LVLMs. Then, we highlight the long-term decay in RoPE [61] (33,, which benefits language modeling but is under-explored for
3

---

multimodal alignment. Finally, we examine the role of RoPE in LVLM object hallucination through comparative experiments, which forms a strong foundation of our design.
LVLM. Typically, an LVLM F is composed of pretrained vision encoder Fu,, a large language model Fi and a projector module f that maps visual embeddings textual space. Given image input T and instruction input 4 (c.g., "please describe this image in detail") "). J encodes these two inputs into multimodal sequence S [ Sinstruct where Suision
f(F (L.)) {um H m== and Sinstruct = Fi((z) {wmm J represent visual and instruction tokens of lengths V and T, respectively. In such sequence, visual and instruction tokens share the same dimension noted as Wm E Rd
Rotary Position Encoding in LVLM. In LLMs like LLaMA [63 and its multimodal successors,
RoPE [6] encodes position information with input tokens by multiplying every token Wm with a rotation matrix Rg)
cos m0
sin m01
sin m0l
cos m0
0
cas m0
sin m0
0
Rd.
sin m0l
cos m02
(1)
COS md
sin mm cos mf a/2
sin ml a/2
where m ..., V + T] indicates position of input token wm and {0, 10000-2(--1)0) 1,22,..,d/20 d/2)) are pre-defined sinusoidal function values following (65] In LVLMs like LLaVA (46), rotary matrices R2 are applied to query and key tokens in all decoder layers, such that relative position dependency among tokens are modeled and integrated in self-attentions across the network. In comparison absolute position encodings [65] and learnable position encodings in ViT (T8),
RoPE captures relative distance among input tokens and has the potential to extend the input context window beyond fixed length (53).
RoPE Long-term Decay. Assume a query token qi at position i and a key token ky at position j. which are derived from input tokens w;, Wj. The self attention 0z,j between tokens qi and ky can be calculated via
Ky
ai.j = softmax
(2)
RoPE applies rotation matrix Rd. to the self-attention above, which is in the form of,
of -(R RØo kj softmax Vd
of
R kj
az,j softmax
Vd
where stands for relative position between @i and ky. The long-term decay refers to the decrease of 84.j as the relative distance increases. As presented in Fig. (c), visual-to-inssruction information flow (i.e., instruction-to-viiuaa self-attention) less significant when j i large and vice versa.
This is favorable for pre-trainee tre-trained LLMs like LLaMA 63 as it aligns with language modeling intuition: pairs of tokens with a long relative distance should have weaker connection. However, we observe that this property brings negative effect in multimodal alignment, in which case visual tokens far from instructions are less attended. This is not expected for multimodal alignment, as the connection between instruction tokens and visual tokens should not be attenuated by their relative distances.
Pilot Experiment. We quantitatively examine the effect of RoPE long-term decay on LVLM object hallucination. To determine how object hallucination is influenced by the distance between visual and instruction tokens, we first train two LVLMs following [46] with two different position alignment strategies, including:
TTraining details for these two models are in Appendix
C.1

---

(a) Aggregated correct responses with FP baseline raster scan
(b) Aggregated correct responses with JFT reverse raster scan
2013
1070
1121
1143
1160
1181
11.99
1207
1191
1193
1123
1153
1226
1297
1345
1364
1371
1380
1334
1400
1389
3400
1526
1283
1202
1226
1283
1325
1206
1304
1295
1345
1347
1339
1264
1226
1363
1359
1400
1408
1379
1395
1367
1429
1427
1416
1345
1331
1360
1353
1334
1435
3442
1366
1354
1425
146
1357
1323
1325
1340
1323
1339
1355
1426
1473
1462
1472
1447
1345
1359
1368
1329
1299
1322
1352
1374
1415
M45
1476
1399
1485
1426
1434
1414
1389
1333
1288
1340
1425
3416
1396
1366
1426
1379
1434
1380
1352
1287
1263
1304
1387
1370
1343
1315
1425
3405
1390
1337
1317
1415
3414
1327
1321
1379
1357
1368
1317
1328
1300
1273
1289
1365
1394
3317
1283
1336
1339
1287
1276
1284
1349
1356
1306
1270
1342
1368
1331
1312
1337
1333
1364
1374
1325
1301
1352
1396
1366
1387
1351
1224
1256
1201
1275
1225
1215
1231
1252
1200
1282
1272
1223
1500
3486
1518
1503
1522
1511
1483
1477
1524
3512
1477
1469
1364
1371
1373
1375
1309
1362
1325
1353
1306
1306
1340
1294
1504
1383
1407
1366
1426
1488
1504
1337
1529
1514
1486
1499
1548
1530
1452
1366
1294
1361
1357
1358
1366
1359
1361
1356
1238
1196
1198
1334
1361
1362
1325
1365
1327
1326
1256
1250
1154
11:35
1360
1216
1208
1217
1156
1190
1214
1213
1185
1149
1091
Figure 2: Motivation Experiment. Given an image L. with object Oz., we crop O,, and paste to various spatial positions {VI, ..., U& within pre-defined template. For every pasting position, we ask two LVLMs (F, and FF) if object O, is this template, where F, refers to a baseline model that follows raster-scan positional alignment strategy and Fr refers to model that resorts to reversal raster-scan position alignment strategy. The total number of correct responses at different pasting positions {ui...., } is reported in (a) and (b), which refers to results from model F, and Fr. respectively. We observe that LVLM Fs are more likely to generate correct responses when pasting object O₂ to lower region, while F, are less hallucinated when pasting object O₂ to upper region. Pasting positions with the most and the least correct responses are highlighted in solid-line and dotted-line red boxes. More details are provided in Appendix C.I Best viewed in color.
F (raster-scan): it follows 46) the position alignment strategy on visual tokens Spision- Under this scenario, visual tokens follow a sequential order, starting from upper-left corner to lower-right corner of input 2-D visual features, row by row. The order of a multimodal sequence S is in format of {1, 2,.,, V.V + 1, ..., V T
F" (reverse raster-scan): it reverses the sequential order of visual tokens Suision- In this case, sequence order of visual tokens starts from lower-right corner of input 2-D visual features to lpper-left corner, row by row. The order of full multimodal sequence S is in format of (V, V 1,.... 1, V + 1,... V + T).
The reverse raster-scan model F alters relative positions between visual tokens Smision and instruc- tion tokens Sinstruct- For example, for instruction token wy+ its relative distance to visual token wy changes from 1 to V, resulting in weaker correlations between wy and wV+1-
Our experiment setup is as follows. Given an image L,, we follow [1] and ask questions in polling-base manner. Specifically, for an object O, in image L., we follow the instruction format of "is there a/an (object) in this image?" to test our models. We crop region of object O2 from I according to its bounding box annotation and paste the cropped object over different positions of a pre-defined image template (more details are covered in Appendix C.T.. This results in new images (To.: ..., ], where (#1,.., U& indicates different pasting positions. We carry out these testing over N images from [42 and aggregate correct responses with respect to pasting positions {"1,... )x
The quantitative results of model J and J are visualized in Fig. 2 (a) and (b), respectively. For model Fb, we find that the response is less likely correct when object O, is pasted on the upper part of the image, and it is more likely correct when object O is pasted on the lower part of image template. This is in stark contrast to F experimental results: model responses more likely to be correct when pasting image crop O,, the upper part of images, while less likely to be correct when pasting position is the lower part. For model Fr, we note
RoPE affects object hallucination.
For demonstration purpose, we assume visual tokens are pre-pended before instruction tokens. For imple- mentation, adapt design for flexible structure of multimodal sequences.
5

---

that visual tokens of lower part is far from instruction tokens in relative distance, corresponding to worse performance in object hallucination. We can thus conclude that RoPE long-term decay affects object hallucination for L.LMMs, which requires special care to mitigate this issue.
4 Concentric Causal Attention
To this end, we introduce Concentric Causal Attention, simple position alignment strategy that mitigates object hallucination by tackling the long-term decay issue originated from RoPE. Our methodology is guided by two key principles.
Alleviate the effect of long term decay on object hallucination by minimising overall relative distance between visual tokens Spision and instruction tokens Sinstruct-
Mitigate performance discrepancy between raster scan model F and reverse raster scan model F
(a) raster-scan positional assignment
(c) raster-scan causal attention mask
JB-1
V+++2
2:8 1 2/7
W
vp
T
Ve 2
0
2,70 +10
208
10
+1
(b) concentric positional assignment
(d) concentric causal attention mask
1
1
2
2
1
v8
/
1
2
1
1
Figure 3: An overview for Concentric Causal Attention. Left: Visual Token Re-organization. In comparison to raster-scan positional alignment in (a), we design concentric position alignment in (b) which shortens visual- risual-instruction distance and retains spatial locality for 2-D data like images. Right: Concentric Causal Masking. By default as in (c), a visual token attends to all preceding visual tokens in 1-D sequence. In contrast, concentric causal attention in (d) models 2-1) continuous positional dependencies among visual tokens, where center visual tokens attend to peripheral ones. Causal masks V by V where this case V 36 for demonstration purpose. Best viewed in color.
Concentric Positions. In existing LVLMS such LLaVA 46 visual tokens are perceived in 1-D continuous sequence (raster-scan position alignment as illustrated in Fig. 3 (a)) and concatenated with instruction tokens for multimodal alignment. We note that such row-by-row positional alignment strategy is not natural for 2-D image data, as it breaks spatial continuity on column dimension. Due to the long-term decay in RoPE, information flow from visual token Wim to Wm++1 differs from that to w. m++ vV. which diverges from spatial locality of 2-D visual features.
Instead of adopting raster-scan sequential order, we design concentric positional alignment strategy as illustrated in Fig. 3 (b). our design, position m of visual tokens are organized form of 2D concentric square, which increases from the peripheral of 2-D inputs to the center. In comparison to sequence order [1, 2, ..., V) for visual tokens Spision such concentric positional alignment reduces relative distance between visual and instruction tokens Sinstruct. For visual token sequence

---

of length V and a instruction token sequence of length T, the maximum distance between visual tokens Svision and instruction tokens Sinstruct is VV + T 1). This concentric sequential ordering also better maintains 2-D spatial locality of visual tokens. Under this scenario, visual tokens that are closer in euclidean distances are causally correlated when position m increases. Meanwhile, visual tokens that share the same position are correlated in visual self-attention. We note that such design mitigates negative effect from RoPE long-term tong-term decay, via decreasing relative distances between Spision and Sinstruct while keeping causal inference scheme in pre-trained L.Mss like LLaMA 64
Concentric Causal Masking. Another part of our method resorts to modification of default causal attention masking towards our concentric visual token reorganization. As presented in Fig. 3 query feature qm (derived from Wm) only attends preceding key features <<=m. Likewise for our method, we follow the same principle to force causal attention masking in 2-D visual inputs. We visualize our masking in Fig. (d), where the total length of visual tokens are 36 (6 by 6). Combining visual token re-oranizzation with concentric causal masking, our method models 2-D continuity for visual inputs and effectively mitigates the object hallucination issue brought by long-tern decay in RoPE.
(c), a
5 Experiments
We first describe training details for our position alignment approach and evaluation setups in Sec.p.11 Subsequently, we report results for several popular benchmarks that demonstrates efficacy of our simple design in the remaining subsections. Further, we present qualitative comparison in Appendix D.2 where our approach generates less hallucinated responses. From a broader scope, we present that our positional alignment strategy benefits general perception capability of LVLMs, where preliminary experiments show that it surpasses the baseline consistently over six multimodal We refer to these results in Appendix D. due to page limits. By default, we conduct our training and evaluation with Vicuna-7B [11] model, unless otherwise stated.
benchmarks [6 28 22 48 49
5.1 Training Details
we adopt pre-trained CLIP ViT-L/14 (55) with 336x336 resolutions as visual encoder and Vieuna-7B [2] as LLM, and 2-layer MLP that connects the visual encoder and LLM. Training consists of two stages, including 1) pre-training over CC-558K dataset [46] with global batch size of 256 and 2) instruction tuning with a 665k multi-turn conversation dataset 45] with
Following [46 45
global batch size of 128.
5.2 POPE
Polling- based Object Probing Evaluation (POPE) [1 is proposed to provide a detailed evaluation of object hallucination in LVLMs, by querying the models about presence of specific objects in given images with yes-or-no questions. POPE adopts three sampling options to sample negative
objects: random, popular and adversarial. We refer to [1 for these setups. Following [4 three
datasets are included in our evaluation, including COCO 42 GQA (28) and A-OKVQA [58]. For
each evaluation setup, every subset includes 3,000 questions for 500 images, which leads 27,000 yes-or-no questions in total.
The experimental results are presented in Tab. Our method achieves the highest accuracy and Fl scores across all datasets and negative sampling setups. By re-organization of visual tokens and concentric masking, our approach achieves 5.48%, 7.86% and 6.70% accuracy improvement and 5.89%. 7.71% and 19% Fl score improvement over the baseline model [46]. We also observe consistent and notable performance gains against state-of-the-art hallucination mitigation methods. CCA surpasses VCD 34 by 1. 1.02%, 4.51 % and 2.65% on three datasets. Particularly, we observe 3.09%. 5.01% and 3.59% Fl score improvement over adversarial evaluation set, which selects the most frequent co-occuring objects with ground-trute objects in image inputs, posing challenges for LVLMs to discern spurious correlation. Our trained model is also comparable to LLaVA-RLHIT model (with Vicuna-13B as its LLM) (22 that additionally aligns model responses with human preference. These results indicate importance of re-organizating visual tokens in vision-language alignment.
7

---

Table 1: POPE Results. acc: accuracy. fl: fl score, measured by precision and recall. Baseline and VCD results are reported by paper 34
random
popular
adversarial
avvrrage
Evaluation
Method
acc
fI
acc
f1
acc
fl
acc
fl
haseline
83.29
81.33
81.88
80.06
78.96
77.57
81.38
79.65
VCD (34
87.73
87.16
85.38
85.06
80.88
81.33
84,66
84.52
MSCOCO
22
LLVAA-RLII
69
85,90
83.92
83,90
82.05
82.60
80.88
84.
82.28
CCA-LLaVA
88.03
86.65
86.87
85,54
85.67
84.42
86.86
85,54
buseline
83,45
82.56
79.90
79.59
74.04
75.15
79.13
79.10
VCD 34
86,15
86.34
81.85
82.82
74.97
77.73
80.99
82,30
A-OKVQA
58
LLaVA-RLHE 02
87.67
86,60
85,20
84.34
79.97
79.92
84.28
83.62
CCA-LLavA
90.27
89.71
88.40
87.98
82.30
82.74
86.99
86.81
baseline
83.73
82.95
78.17
78.37
75.08
76.06
78.99
79.13
VCD
86.65
86,99
80.73
82.24
76.09
78.78
81.16
82.67
GQA 28
LLaVA-RLHE 62
84.93
83,38
81.37
80.23
78.30
77.70
81.53
80.44
CCA-LLaVA
88.40
87.68
86.47
85.91
82.20
82.37
85.69
85.32
5.3 CHAIR
We further evaluate our method on Caption Hallucination Assessment with Image Relevance (CHAIR) pioneering study introduced to measure object hallucination in image caption- ing [57] quantifies the factuality of a model by calculating the proportion of objects not present in ground truth over all objects in caption output. It contains both instance level score CHAIR, (shorted for C7) and sentence level score CHAIRs (Cs) which holistically assess model's performance. Specifically, CHAIR metric is formulated
metric. CHAIR
Cs |sentences with hallucinated objects)|
[hallucinated objects}|
C₁ (all mentioned objects)
[all sentences]
where lower scores corresponds to better performance. Following previous studies 26 we prompt LVLMs with "Please describe this image detail.". Note that LVLM's performance on CHAIR metric is highly dependent on their output sentence length. Short and succinct responses have less chances to make mistakes and thus would generally have better CHAIR scores. Different textual prompts such as "in detail" and "in brief" also influences output length and creates bias in CHAIR evaluation 4 To offset the influence of output length and prompt phrasing and ensure fair basis of comparison, follow the experimental setup in OPERA (26) and set the maximum text token to 64 and 512 respectively to examine hallucination on both short and long responses. Following (26), we sample 500 images from COCO VAL 2014 42 generate descriptions from different models and hallucination mitigation methods.
Our image caption evaluation result on CHAIR is shown in Tab. 2 For greedy decoding, our model surpasses baseline model [46] by 3.2% while maintaining high object recall (80.3% v.s. 80.4%) for long-response generation (by setting max new tokens to 512). Note that longer textual responses suggests more significant distance between visual and instruction tokens, leading to higher hallucination rates [33, which can be improved by our approach that reduces relative distance between visual and textual tokens. Our results are comparable against LLaVA- [62] over this setup. On short responses, 0.8% on instance-level while maintaining high object recall.
model also outperforms bascline model by 2.8% on sentence-level and
Our approach is also effective when using beam search for autoregressive decoding. We surpass the baseline by 0.8% and 0.5% on long-response generation, and 2.2% and 0.5% on short-response generation for Cs and C₁, respectively. Our approach is also complementary to OPERA [26]. In comparison to baseline model that using OPERA decoding, our approach are 1.8% and 1.1% better for Css and C on long-response setting. We observe consistent performance gains in rhort-response generation (1.6% for Cs and 0.9% for C7). Quantitative evaluations on open-ended generation indicates importance of better positional alignment strategy and efficacy of our design.
8

---

Table 2: CHAIR results. For evaluation setups, 512 and 64 refer hyperparater that relates the length of LVLM repsonses, corresponding to long-text and short-text generation, respectively.
512
64
Evaluation
Method
C
Cj
len
C
C1
rec
len
baseline
46.2
12,9
80.3
97.2
21.0
6.2
66.3
$4.9
greedy
LLaVA-RLIH RLHF 62
10.5
78.0
117.9
19.6
5,4
64,9
$4.0
CCA-LLaVA
43.0
11.5
80.4
96.6
18.2
5.4
66.7
54.5
bascline
49.4
13.9
79.9
96.1
18.2
64.0
52.7
OPERA
20
46.8
13.4
79.6
93.2
17.8
5.9
64.3
53.0
bean (5)
CCA-LLaVA
48.6
13.4
79.9
94.2
16.0
5.3
64.8
52.7
CCA-LLaVA + OPERA 26
45.0
12.3
79.5
91.8
16.2
5.0
65.0
52,9
5.4
MME
The MME hallucination subset extends scope beyond object hallucination. Following [34]. we evalu- ation perception sub-tasks that examines LVLMs on object-level and attribute-level hallucinations, including measure of object existence, count, position and color. As presented in Tab. our method surpasses the baseline by 76.33 on these tasks. comparison to previous hallucination mitigation method VCD, our approach demonstrates non-negligible performance gains over all subtasks (e.g.. 2.00 improvement from VCD v.s. 24.00 improvement from our method). These results indicate the potential of CCA to improve general perception capability of LVLMs.
Table 3: MME results.
Table 4: LLaVA Bench (In-the-Will results.
Object-level
Attribate-leve
Model
Complex
Detail
Conv
Overall
Model
51.2
OPERA
baseline
175.67
180.67
124.67
133.33
114.00
123.33
155.00
151.00
592.33
565.33
OPERA 26
66.4
$6.9
44.0
61.3
VCD 14
604.66
VCD
69.6
51.8
$7.3
61.6
641.66
66.1
5.5 GPT4V-Aided Evaluation
We also evaluate approach LLaVA-Bench (In-the-Wil)) [46, composed of 24 images with 60 questions in total. LLaVA-Bench (In-the-Wild) constitutes three types of questions, including conversation, detailed description and complex reasoning. Following [46] 26), ask these models to generate responses and let the text-only GPT-4 [ be the judge to rate these responses. The results presented Tab. 4 In comparison to OPERA (26) that specializes in open-ended generation, our
method still stands out when examined by GPT-4 according to detailness and correctness, suggesting efficacy of our positional alignment strategy on generating accurate long responses.
Conclusion and Limitations
In this paper, we aim to mitigate object hallucination in Large Vision-Language Model (LVLM). We perform in-depth analysis on correlation between object hallucination and Rotary Position Encoding, widely used positional dependency modeling design in existing LVLMs. We find that LVLMS are more likely to hallucinate when relevant visual cues are distant from instruction tokens in D multimodal sequence, due long-tern decay in RoPE. To this end, propose Concentric Causal Attention, simple yet effective positional alignment strategy that reduces relative distances between visual and instruction tokens, alleviating negative impact brought by RoPE long decay object hallucination. Experimental results over multiple evaluation benchmarks supports our design, indicating importance of better position alignment strategy.
Limitation. While this study shows improvements on mitigating object hallucination LVLMs, our focus is only limited to handling of image-text inputs. We consider positional alignment strategy for other modalities of input data as future works, such audio or video inputs that differs from image-text modalities.

---

Acknowledgments and Disclosure of Funding
This project is funded by the Ministry of Education Singapore, under Tier- 1 project scheme with project number RG18/22 and Tier-2 project scheme with project number MOE-TTEEP20220-00000
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Lconi Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08774 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yama Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237555 2022.
[3] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, YinnYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406. 12718, 2024.
[4] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308 01390, 2023.
[5] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading. and beyond. arXiv preprint 2rXiv:2402.16050 2023.
[6] Junhum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal Ilm. arXiv preprint arXiv:2312.06742 2023.
[7] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint F:XXiv3306.11990 2023.
[8] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint XxXiv:2403.20330 2024.
[9] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang. Jianfeng Gao, and Chunyuan Li. Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing. arXiv preprint arX00.231..0057. 2023.
[10] Zhaorun Chen, Zhuokai Zhao Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou, Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.004: 2024.
[11] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
[12] Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[13] Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. Visionllama: A unified Il interface for vision tasks. arXiv preprint urXiv:2403.0003 00552, 2024.
[14] Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint vrXiv:2311.032877 2023.
[15] Wenliang Dai. Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung. and Steven Hoi. Instructblip: Towards general-purpos lision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages leee, 2009.
10

---

[17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2022000..19920 /9999, 2020.
[19] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303-14312, 2024.
[20] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[21] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models, arXiv preprint arXiv:2310. 14566, 2023.
[22] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 0008-661,,00 2018.
[23] Pengeheng Hc, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: disentangled attention. arXiv preprint arXiv:2006.0366 2020.
bert with
[24] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735 1780. 1997.
[25] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.090 09685, 2021.
[26] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospetioonallooatioo arXiv preprint 2xXi:2331.179/1 2023.
[27] Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. arXiv preprini arXiv:2009.111 13658, 2020.
[28] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[29] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Yc, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. arXiv preprint arXiv: 2023.
[30] Dongfu Jiang. Xuan He, Huaye Zeng. Con Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis; Interleaved multi image instruction tuning. arXiv preprint arXiv:2405.01453 2024.
[31] Guolin Ke, He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. arXiv preprint 200: 15595, 2020.
[32] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint S3tt::2000006999, 2023.
[33] Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. Volcano: mitigating multimodal hallucina- tion through self-fccdhack guided revision. arXiv preprint arXiv:2311.07362 2023.
[34] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large languue models through visual contrastive decoding. arXiv preprint arXiv:2311. 16922, 2023.
[35] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in co-context instruction tuning. arXiv preprint 30002200000376,,.. 2023.
11

---

[36] Bohao Li. Rui Wang, Guangzhi Wang. Yuying Ge. Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping toaguugge-magee pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19742. PMLR, 2023.
[38] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages I 2888-12900. PMLR, 2022.
[39] Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Myhench: A comprehensive multi-modal video understanding benchmark. arXiv preprint 2023.
[40] Yanwei Li, Yucchen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024.
[41] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- JinRong Wen. Evaluating object ballucination in large visoonllaguage models. arXiv preprint arXiv:2305. 2023.
[42] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Cumputer Vision-ECCV 2014: 13th European Conference, Zurich, Swirzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014,
[43] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang. Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2024.
[44] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang. Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprini arXiv:2402.0000 00253, 2024.
[45] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[47] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.05449, 2023.
[48] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2007.0628 2023.
[49] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Wei Chang. Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-25211. 2022.
[50] Zeyu Lu, Zidong Wang. Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402. /2376, 2024.
[51] Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narrator via equal distance visual tokens. arXiv preprint 2rx.::2312.08870, 2023.
[52] Jiahao Nie, Gongjie Zhang, Wenbin An, Yap-Peng Tan, Alex C Kot, and Shijian Lu. Mmrel: A relation understanding dataset and benchmark the milm era. arXiv preprint arXiv:2406.0912. 2024.
[53] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole, Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0000 2023.
[54] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.1483 14824, 2023.
12

---

[55] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[57] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Bums, Trevor Darrell, and Kate Sacnko. Object hallucination in image captioning. arXiv preprint arXiv: 1809. 02. /56, 2018.
[58] Dustin Schwenk. Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 162. Springer, 2022.
[59] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.00 02055.0...22.. 2018.
[60] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317 8317-8326, 2019.
[61] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position emhedding. Neurocomputing, 568: 127063, 2024.
[62] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 1iXi:23339.1525555 2023.
[63] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[64] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv: 092888 2023.
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[66] Junyang Wang, Yuhang Wang. Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang, An llm-free multi-dimennional benehmark for milmss hallucination evaluation. arXiv preprint arXiv:2311.0783 2023.
[67] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint L.X30923/..0309, 2023.
[68] Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jachong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401. 105. 2024.
[69] Yuxuan Wang, Yueqian Wang, Pengfei Wu, Jianxin Liang. Dongyan Zhao, and Zilong Zheng. Lstp: Language-guided spatial-temporal prompt leaming for long-form video-text understanding. arXiv preprint arXiv:2402. 16050. 2024.
[70] Junfei Wu, Qiang Liu, Ding Wang. Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncovering object hallucinations in large vision-language models. arXiv preprint arXiv:2402. 16222 2024.
[71] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[72] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint 2rx.v:2306.13549, 2023.
13

---

[73] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu. Hao Wang, Dianbo Sui. Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310. 16045, 2023.
[74] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang. Zirui Wang. Liangliang Cao, Shih-Fu Chang. and Yinfei Yang. Ferret: Refer and ground anything any at any granularity. arXiv preprint arXiv:2310.07704, 2023.
[75] Qifan Yu, Juncheng Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang. Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data, 2023.
[76] Tianyu Yuan Yao, Haoye Zhang, Taiwen He. Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng. Maosong Sun, et al. Rlhf-v: Towards trustworthy milmss via behavior alignment from fine-grained correctional human feedback, arXiv preprint arXiv:23
2023.
[77] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning, arXiv preprint WxXi:2312.100333 2023.
[78] Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective, 2024.
[79] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 028588 2023.
[80] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772 2024.
[81] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-intersst arXiv preprint 2307.0307..3601 2023.
[82] Yuechen Zhang, Shengju Qian, Bohao Peng. Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for modli Ilms. arXiv preprint arXiv:2312.04302. 2023.
[83] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision lssionllaguuag models. In The Twelfth International Conference on Learning Representations, 2024.
[84] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny. Minigpt-4: Enhancing vision- linguae understanding with advanced large language models. arXiv preprint arXiv:2304.1054) 2023.
[85] Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self-supervised visual preference alignment. arXiv preprint ar 2404 10501, 2024.
14

---

Appendix
A
Broader Impact
Like other LVLMs, models trained by our CCA approach have their potential benefits and risks when they are publicly released. As approach is validated on LLLVV which constitutes CLIP,, Vicuna and LLaMA, trained models may inherit risks from these pre-trained visual encoders and large language models, including handling malicious inputs, hallucination or potential biases. We mitigate these issues following other LVLMs.
B RoPE in LLaMA
We further clarity the role of Rotary Position Encoding (RoPE) [6] LLaMA architecture with separate illustration. As Fig.4 shows. RoPE densely involved LLaMA [63 64]. namely all self-attention layers. This architectually distinct from how positions are involved ViT, where absolute PEs are only added once right after patch embedding layer. As most open-source LVLMs are using LLaMA as their language it is noteworthy to study how RoPE may affect multimodal pereeption when connect pretrained vision models (e.g., CLIP) with LLaMA.
backbones [46 84 13 67
30 79
Query Key Feature Transformation via RoPE
LLaMA Overview
Inputs
RMS Norm
RoPE
Nx
Self-Attention
RMS Norm
Feed Forward
RMS Norm
Linear
o
D
outputs
Figure 4: RoPE LLaMA. schematic view for LLaMA where RoPE highlighted, and an example illustration how RoPE is applied over query key feature. We use short input sequence with length of 4 and feature dimension of 4 for demonstration purpose. Input tokens are rotated with angles, subject token positions. For mathematical definition, please refer to Sec. 3
C Implementation Details
We include more details here about implementaioon for Fig. Fig. [and Fig. reeuulsss in main paper, including data and model architecture we use, and training details we follow.
C.1 Pilot Experiment
Training. As described in Sec. Sf main paper, we train baseline LVLM F& that follows raster-scan positional alignment and another LVLM F, that follows reversal raster-scan position alignment. For these two models, we carry out two-stage training following [46), except for the second stage we train both models for 20K steps with LoRA (25) due to resource limitations. Both experiments share the same training hyper-parameters 665K full schedule training.
motivation experiments. each annotation with its corresponding image, we crop an object according to bounding box and paste within a pre-defined template (a visually gray image), which is initialized with ImageNet [16] average pixel
Inference. We sample 3,000 annotations from COCO VAL 2014 42
carry
values. We test & spatial positions (U1,.., vx where & is set to 144, resulting in resolution of 12 by 12 for both aggregated results in Fig Workflow on how construct such synthetic data is further presented in Fig. 5
15

---

Image
Synthesized Image
Crop & Paste
Cropped region
Figure 5: Workflow illustration on how we synthesize testing data. Given an image and box annotation for one object instance, we crop it and paste on a template image, initialized with ImageNet mean pixel values. We paste every cropped region every spatial position. Resulting data constitutes a large amount of questions about object existence, diverse in spatial positions.
C.2 Information Flow
We reveal long-term decay property RoPE [1] in scope of LVLMs. To implement this, we use 3,000 image- query pairs from POPE [1I] adversarial setup, and LLaVA-1.5-7B 44 as our LVLM. For each image- -uerry pair, we extract and aggregate self- attentions from the first decoder layer of LLaMA (64) We average obtained self-attentions across heads and images obtain quantitative results in Fig. D A pseudo code is provided below for further clarification.
def compute inst Blow
Long
token pos,
img token len
Return
information flow from viaual (via) to instruction (inat) takens.
Impus
alin - self attentions.
ing token pos where image sequence starts.
ing len sequence length for visual tokens.
onn
last vis attn attm]
img token pos img
+ 1t,
iig token pos: iig token pos ling token lon
average images. heads, and instruction tokens. via atin mean(din (0, 1, 2)) retuinn vis inst flow.
D
More Results
D.1
Comparison over Multiple-Choice Benchmarks
Beyond the scope of visual hallucination, we consider our proposed positional alignment as a general approach for improving perception capability for LVLMs. We further evaluate our trained model over six benchmarks that examines LVLMs general perception capability, including SEED-Bende [36,, ScienceQA 49). GQA (28). Vizwiz [22]. MMBench 48 and MMStar [8] which evaluates LVLMs perception capability with multiple choice do our comparison.
questions. We Imms-cval 80
For details of evaluation benchmarks, SEED-Bench (36) consists of 19k multiple choice questions with human annotations, while spanning 12 evaluation dimensions, including both image and video data. MMBench (48 also examines LVLMs on general perception capabilities using wide range of tasks. We also present our comparisons ScienceQA [4)) Vizwiz 22 and GQA 28 that examines certain perception capability. like
iniipeiieiieeesiblle benchmark, which requires better visual comparison to our baseline model LLaVA, our positional alignment approach achieves non non-negligible gains across all six benchmarks, without introducing additional computation during training. Such performance gains highlight potential of Concentric Causal Attention enhancing general visual perception capability of LVLMs.
knowledge and relation. Note that MMStar g
grounding trained LVLMs. We present our results against baseline model [46] Tab,
16

---

Table 5: Evaluation Multiple-Choiee Benchmarks. Baseline results reported by paper (45]. except for MMStar reported by [8] SEED SEED' and SEED" refers to all, image and video evaluation, respectively. SeVa results are reported by 85
Method
SEED 36
SEED 36
SEED 36
SQA 49
GQA 28
VizWiz
MMBench 48
MMStar
TexVVQ
28
3
60
LLaVA
58.6
66.1
37.3
66.8
62.0
50.0
30.0
58.2
LLaVA w/ VCD
58.3
63.7
37.6
68.5
61.9
50.5
34.6
54.4
Seva- 7h-dif
65.8
67.5
60.7
65.6
Seva
67.1
60.9
VCA-LLaVA (ours)
61.7
67.1
41.0
69.8
63.1
57.6
65.4
33.2
57.8
D.2 Qualitative Comparison
We present qualitative comparison between responses generated by baseline model 46 and our trained model. We show that baseline model cup Fig. 7 We also show case study on LLaVA -Bnnch [46] as illustrated Fig. where baseline model hallucinates object hat. We also note baseline model hallucinates optical character recognition and numbers as in Fig. where our method mitigates these issues.
likely hallucinate, for example, bathtub and sink in Fig. knife and
User: Please describe this image in detail.
The image features small cat sitting top white toilet a bathroom. The cat appears be curiously observing surroundings. The toilet situated next bathtub, there shower curtain the background. Additionally, there is sink the bathroom, located towards the right side of the scene.
LLaVA-1.3 7B:
CCA-LLaVA-1.51 7B: The image features small cat sitting on the edge of white toilet bathroom. The cat oppears to be curiously observing its surroundings. The toilet takes up significant portion of the scene, extending from the left the right side of the image.
Figure 6: Qualitative comparison of open-ended generation between baseline and our method.
User: Please describe this image detail.
LLAVA-1.5-7B: image features delicious pizza sitting a on dining table. The pizza topped with tomatoes and cheese, making it appetizing sight. Next to pizza, there is bottle beer, adding to the enjoyment of the meal. Addition to pizza and beer, there is knife the table, likely used for cutting the pizza into slices, A cup can also be seen on the table, passibly containing a beveroge to accompany meal.
CCA (CA-LLaVA-115 5-77: LLLVAA-V----B ready to be eaten. The pizza is topped with tomatoes and cheese, making a delicious and oppetizing meal, Beside the pizza, there bottle of beer, adding to the enjoyment the meal, The pizza beer are placed on dining table, creating cozy inviting atmosphere.
The image features large pizza sitting on top of a cardboard box,
Figure 7: Qualitative comparison of open-ended generation between bascline and our method.
17

---

E
Computation Resources
We use 4 NVIDIA RTX A6000s to train our models. Pre-training stage takes 12 hours while instruction tuning takes roughly 83 hours.
User: What might be the intended effect of this painting
LLVVA-1.5-7B The intended effect of this painting is to create humorous and whimsical scene by combining dog with a human-like appearance, such Rencissance-sttle painting. The and a Renaissance-styl dress, which adds a touch of humor and creativity the image. The painting might be intended to evoke amusement curiasity the viewer, manner. The artist might have wanted to challenge the viewer's expectations and create a unique, memorable piece of ort that stands out from traditional representations dags.
wearing hat
dog dressed in
CCA-LLaVA-1.3 58:78: The intended effect this painting to create a humorous and unexpected twist classic artwork, such the Mona Lisa. painting dog weaning bandona and robe, the artist has combined elements of human art and culture with the presence of a dog, which is not typical subject traditional art. This juxtaposition of humon and canine elements creates lightheorted and amusing scene that deviates from the seriousness formality often associated with classical art. The painting might be intended evoke loughter, surprise, or even serse of playfulress among viewers.
Figure 8: Case Study where question is sampled from LLaVA-Bench [46]. LLaVA hallucinates hat in its long response, while CCA answers correctly without hallucination.
User: What the brand of the yogurt flavored with blueberry?
User: How many uncut fruits are the image?
LLaVA-1.5 5-78:
The yogurt flavored with blusberry mode by the brand Chobari.
LLAVA-155- 5-578: There are four uncuf fruits the image.
CCA-LLaVA-1 1--4--155-700 5-78:
The brand of the flovored with blusbemyy
CCA-LLeVA-1 5-7B: There three unout fruit in the
Figure Case Study where question is sampled from LLaVA- -ench [46]. CCA-LLaVA outperforms LLa VA optical character recognition (left) and numerical prediction in given cases.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: abstract and introduction clearly state the claims made the paper. The claims match experimental results and reflect that our method can he expected to generalize to other settings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Please refer to Sec. limitations.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
19

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Sec. 5.1 in the main text and Sec. C.I for training our models. Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution the contribution is specific model empirical evaluation, it may necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, which case authors welcome to describe the particular way they provide closcd-soure source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
reprodueibility. In the case of
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [Yes]
Justification: We refer readers hotttess/ggthhb..oo///ng//////// for details of data and code guideline.
Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com wwtp:ipppppccppppppppppp Guides/CodeSuumissti osiolollccy for more details.
While we encourage release of code and data, we understand that might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https: /ttiss////ps.../ cc/public/ guiies/Coaessbmiisiioooooey for more details.
20

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer to See. 5.1 in the main text for training details while Sec. 5 the main text and Appendix D.I for evaluation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Dnes the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported since training of LVLMs is computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please refer to Appendix E for GPUs we use and execution time for training LVLMs. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics https://neuripss http:///neurppsccpppliiilieeee.e
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that roquire a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please find Broader Impacts in Appendix A
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality generative models could used to generate deepfakes for disinformation. other hand, it not needed point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the tcchnology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We use open-sourced models and data only. We have properly cited original papers of our training and evaluation data. The license for assets used this paper are under CC-BY 4.0, Our models in this paper will be under CC-BY-NC-SA 4.0 license.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use sual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model implementing safety filters.
22

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited original papers of our training and evaluation data. The licenses for models use include CLIP, which is under MIT License, and LLaMA2, which is under Apache- 2.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
authors should state which version the asset is used and. possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper swi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
24


Reviews:
[{"review_id": "mzSNCv7LVB", "reviewer": "Reviewer_9xbc", "summary": "This paper explores how current LVLM's hallucination appears through analyzing the impact of RoPE long-term decay on vision information  attenuation during flow. It gives clear visualization results and theoretical evidence to prove the basic point, revealing that the causal attention mask and RoPE embbeding is not appropriate for non-text-modal input (e.g., vision tokens). Based on these, the authors argue that LVLM's hallucinations are mainly attributed by the RoPE long-term decay and the mismatch between vision tokens and causal attentions. To this end, the authors propose vision token re-organization and concentric causal attention masking to alleviate the hallucination. Experiments demonstrate the promising performance of the proposed methods.", "questions": "See the Weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper gives a good explanation for the relationship between hallucination and long-term decay in LVLMs.\n2. The proposed method is well motivated and technically sound.", "weaknesses": "1. When introducing the long-term decay and the attenuation of information flow, it would be better to cite some references like OPERA (some of the early works that claim the relationship between hallucination and long-term decay).\n2. One suggestion for the experiment on Figure 2: Although the authors calculate and visualize the results on thousands of samples, it would be better to reverse the order of vision patches (before CLIP-ViT) and conduct the same experiment again. We should rule out the possibility that most of correct answers are naturally located at the lower region of images.\n3. Both the proposed method relies on the hypothesis that the most of main contents of images will be located at the centric region. It generally make sense, but not enough to be an accurate solution.\n4. The experiments are somehow insufficient. It would be better to add some ablation studies and evaluation on LVLM benchmarks. MME hallucination split and LLaVA Bench are not enough. It would be better to add results on MMBench, Seed Bench, TextVQA, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and thorough suggestions. Please find our replies as below.\n\n**W1: CCA and OPERA [23].**\n\n**A:** We ground our design on analysis of information flow in LLaVA model. This shares commonalities with OPERA which analyzes information flow in LVLM autoregressive decoding. Thank you for pointing out and we would highlight this in introduction of our manuscript as suggested. Different from OPERA that discovers co-occurrences of aggregation pattern and object hallucination, our CCA explores relations between Rotary Position Encoding and object hallucination. We further refer to quantitative results in Table 2 and 4 of our manuscript, where proposed CCA surpasses OPERA in CHAIR and LLaVA-Bench evaluations.\n\n**W2: Figure 2 experiments.**\n\n**A:** It is a valid concern to negate the impact of imbalanced object distribution in different image regions. We kindly let you know that we already addressed this in the paper (lines 167-169) by cropping objects and pasting them on blank images (initialized with ImageNet mean pixel values) at different spatial positions to create synthesized images. Please find a new illustration for this in `Figure 4` of uploaded `pdf`, where 16 (4 by 4) pasting options are given. In our experiments we use 144 (12 by 12) pasting options. Figure 2 of our manuscript is obtained by testing models on these synthesized images which have even distribution of objects at different regions. \n\n**W3: Data statistics and our concentric design.**\n\n**A:** Yes, our concentric design assumes that most image contents are located around the centric image region. We validate this assumption from two perspectives.\n\nFirstly, we perform statistical analysis on large amount of natural images (82,081 images and 604,907 annotations from COCO train 2014, and 10,696 images and 174,304 annotations from GQA). Specifically, we count total number of objects in 9 spatial locations (top_left, top_mid, top_right, mid_left, center, mid_right, bottom_left, bottom_mid, bottom_right, respectively). The statistical results in `Figure 3` of uploaded `pdf` show that more objects are located in centre, which aligns with our design. \n\nSecondly, we point out that for our model, positions start from periphery of 2-D visual tokens and ends in centre. For comparison, we implement another LVLM (which we name as CCA-r), where positions start from centre of 2-D visual tokens and ends in periphery. Please find an illustration of CCA-r in `Figure 2` of uploaded `pdf`, where left refers to our CCA method and right refers to CCA-r. Query tokens, key tokens for self-attention calculations are highlighted with colors. \n\nThe tables below provides quantitative experiments on POPE and CHAIR, showing that changing from CCA to CCA-r positional alignment causes performance drop in most evaluations. For GQA popular evaluation, accuracy drops from 88.40 to 83.43. For CHAIR evaluation, c_s drops from 43.0 to 50.0. These validate the assumption that most image contents are located around centric image regions, which aligns with our design.\n\n|**POPE**|ran||pop||adv||\n| - | - | - | - | - | - | - |\n||acc|f1|acc|f1|acc|f1|\n|**MSCOCO**||||             |               |               |\n| CCA-r (new) | 87.43    | 85.90       | 86.33       | 84.85       | 85.17         | 83.77         |\n| ours           | **88.03**     | **86.65**   | **86.87**   | **85.54**   | **85.67**     | **84.42**     |\n| **GQA**        |               |             |             |             |               |               |\n| CCA-r (new) | 88.63    | 87.99       | 83.43       | 83.41       | 81.83         | 82.09         |\n| ours           | **90.27**     | **89.71**   | **88.40**   | **87.98**   | **82.30**     | **82.74**     |\n| **A-OKVQA**    |               |             |             |             |               |               |\n| CCA-r (new) | **89.70**   | **89.12**       | **86.90**       | **86.56**       | 81.20         | 81.78         |\n| ours           | 88.40         | 87.68       | 86.47       | 85.91       | **82.20**         | **82.37**         |\n\n|**CHAIR**|512||||64||||\n|-|-|-|-|-|-|-|-|-|\n||c_s (↓)|c_i (↓)|rec|len|c_s (↓)|c_i (↓)|rec|len|\n|CCA-r (new)|50.0|18.1|**85.7**|96.4|**18.0**|5.5|66.1|54.7|\n|CCA|**43.0**|**11.5**|80.4|96.6|18.2|**5.4**|**66.7**|54.5|\n\n**W4: More evaluations on LVLM benchmarks.**\n\n**A:** Please refer to Table 5 in Appendix C.2 where we included more evaluations on LVLM benchmarks. As suggested, we also add TextVQA [A] for your reference. We also compare general perception capabilities on these benchmarks against two hallucination-mitigating methods [30, B], where SeVa [B] is a recent method that explores unsupervised preference alignment in LVLMs. We point out that SeVa trains their models with another Direct-Preference-Optimization stage on top of LLaVA-1.5 [41] whereas our CCA does not involve any new training stages and strictly follows LLaVA-1.5 training scheme. Our model LLaVA-1.5-7B-CCA outperforms SeVa-7B and VCD on most LVLM benchmarks.\n\n| Model                          | SEED-all | SEED-img | SEED-vid | SQA | GQA  | VizWiz | MMBench | MMStar | TextVQA |\n| - | - | - | - | - | - | - | - | - | - |\n| llava-1.5-7b                   | 58.6     | 66.1       | 37.3       | 66.8       | 62.0 | 50.0   | 64.3    | 30.0   | **58.2**    |\n| vcd-llava-1.5-7b (new) | 58.3     | 63.7       | 37.6       | 68.5       | 61.9 | 50.5   | -       | **34.6**   | 54.4    |\n| seva-7b-diffu500 (new) | -        | 65.8       | -          | 67.4       | 61.1 | -      | 64.7    | -      | -       |\n| seva-7b-diffu800 (new) | -        | 65.8       | -          | 67.5       | 60.7 | -      | **65.6**    | -      | -       |\n| seva-7b-moco (new)     | -        | 65.5       | -          | 67.1       | 60.9 | -      | 65.2    | -      | -       |\n| llava-1.5-7b-cca (ours)        | **61.7**     | **67.1**       | **41.0**       | **69.8**       | **63.1** | **57.6**   | 65.4    | 33.2   | 57.8    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBJFtwKe69", "reviewer": "Reviewer_1NX2", "summary": "This paper analyze the long-term dependency between text token and visual token in LVLMs from a novel positional encoding perspective by replace the RoPE method. The analysis shows that RoPE introduce clear long-term decay regarding the attention scores. The authors propose a novel concentric causal attention (CCA) including visual token re-organization and concentric causal masking to keep 2-D spatial locality while shortens visual-instruction distance. Experimental results show CCA can reduce hallucination on both discriminative and generative benchmarks while keep comparable performance on general benchmarks.", "questions": "- In Figure 1  (b), would the removal of the RoPE positional encoding makes the self-attention process out of distribution? Since the model is pre-trained with RoPE applied. How would this affect the quantitative results?\n- The explanation of concentric causal masking in Figure 3 and the corresponding paragraph is somewhat unclear. Could you clarify why the attention masks for the first seven rows are identical? A more comprehensive and detail introduction helps a lot.", "limitations": "authors adequately addressed the limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation of exploring source of hallucination from a long-term decay perspective is clear.\n- The paper is well organized and clear.\n- The proposed CCA method seems effective and easy-to-implement.", "weaknesses": "- The experimental setup lacks rigor. The VCD results are sourced from the original paper, which is based on LLaVA 1.5. However, the baseline provided by the authors utilizes the pre-training scheme of LLaVA 1.0 along with the projection module and instruction fine-tuning data from LLaVA 1.5. This discrepancy raises concerns about the comparability of the results. Additionally, it is unclear why Table 3 does not include results for opera and why Table 4 omits the VCD results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and insightful suggestions. Please find our responses as follows.\n\n**W1-a: Pretraining setup.**\n\n**A:** Thanks for mentioning this concern. We would share that it is a typo in line 227 where we claim we use CC-595K dataset [42] for pre-training stage. In fact, our pre-training experiments follow LLaVA 1.5 [41] and use a 558K dataset for pre-training. The provided baseline results in Table 1 of our manuscript is for LLaVA 1.5, which are sourced from VCD paper [30]. We will release our model and source code for the community to reproduce our results. \n\n**W1-b: Manuscript Table 3 and Table 4.**\n\n**A:** Thanks for pointing this out. Please find new quantitative results of Table 3 and 4 below. We use their official codes and get these results.\n\n|MME | Existence | Count   | Position | Color  | Total  |\n|-|-|-|-|-|-|\n| baseline          | 175.67    | 124.67  | 114.00   | 151.00 | 565.33 |\n| OPERA (new)| 180.67    | 133.33  | 123.33   | 155.00 | 592.33 |\n| VCD               | 184.66    | 138.33  | **128.67** | 153.00 | 604.66 |\n| ours              | **190.00** | **148.33** | 128.33   | **175.00** | **641.66** |\n\n|LLaVA-Bench | Complex | Detail | Conv  |\n|-|-|-|-|\n| baseline          | 65.8    | 51.2   | 54.6  |\n| OPERA             | 66.4    | **56.9**   | 44.0  |\n| VCD (new result)  | **69.6**    | 51.8   | 57.3  |\n| ours              | 66.1 | 53.9 | **69.4**|\n\n**Q1: Removal of RoPE from LLaVA.**\n\n**A:** Yes, removing RoPE largely diverges from LLaMA pre-training and leads to nonsense outputs. Our earlier studies showed that LVLMs without RoPE will no longer follow instructions. Take POPE [37] questions as an example. The LLaVA-v1.5-7B pretrained without RoPE fails to answer yes or no as illustrated below.\n> USER: Is there a scissors in the image?\n>\n> LLaVA-1.5-7B: No.\n>\n> LLaVA-1.5-7B w/o RoPE: the bear.\\n\\n\\n\\n\\n\\n\\n\\n\n\nSince the LLaVA-1.5-7B w/o RoPE will output neither yes nor no, the quantitative accuracy on POPE should be 0.00. Though the model w/o RoPE in Figure 1.b of our manuscript generates non-sense text outputs, the information flows from visual to text tokens are more evenly distributed. This highlights the long-term decay in RoPE in Figure 1.c, the root cause of information aggregating at image tokens that are closer to text tokens.\n\n**Q2: Clarifications on concentric causal masking.**\n\nThanks for pointing this out. Please find a new illustration in `Figure 1` of uploaded `pdf` that clarifies the proposed concentric causal masking, where query tokens, key tokens attended by query tokens and key tokens not attended by query tokens (tokens not involved in self-attention computation) are colored. Consistent with our manuscript Figure 3, we take 6x6 visual token organization as an example. The design follows the same causal modeling rule of LLaVA (presented in `Figure 1 (left)` of uploaded `pdf`), where query tokens with larger position values attend to key tokens with smaller or equal position values. The first seven rows of manuscript Figure 3 (d) share the same attention masks, corresponding to `Figure 1 (right)` (a) to (g) of uploaded `pdf`, where the position indices of key tokens attended by query tokens are exactly the same.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4WUW1SWUp", "reviewer": "Reviewer_5Msc", "summary": "The paper shows that object hallucination in LVLMs is linked to the commonly adopted Rotary Position Encoding (RoPE) strategies. The long-term decay in RoPE causes hallucinations when important visual tokens are distant from visual instructions. To address this, the authors propose the Concentric Causal Attention strategy to reduce the distance between these tokens. However, experimental results of the proposed method is not promising on several benchmark datasets.", "questions": "- Please clarify what do the orange and yellow colors represent in Figure 3(d)? The explanation of casual masking in Fig.3d is not clear: need guidance on interpreting the concentric causal masking in Figure 3.\n\n- What is the distribution of the aggregated correct responses in Figure 2 with the proposed concentric causal attention? Does the distribution different from the raster scan and reverse raster scan?\n\n- The paper compared with raster-scan as the baseline. But how do you justify that the concentric positional assignment in Figure 3 is the best solution (or is good enough)? Did you consider other scan designs, for example, the diagonal-scan, zig-zag scan design?", "limitations": "Yes, the author adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The analysis of the long-term decay in RoPE and its impact on hallucinations in LVLMs is novel.", "weaknesses": "- The method section is quite short, and it seems not covering a comprehensive solution to address LVLM hallucination problem. It lacks justification on the proposed scanning method. Specifically, the concentric causal masking section is not clear.\n\n- Performance of the proposed method is not promising on several benchmark datasets. For example, the proposed method sometimes achieves the best results only when combined with other state-of-the-art methods. It might provide more insight if there is an analysis on when and why the proposed CCA alone will work or fail (Table 2).\n\n- The detailed description of the method with the figure is not clear. Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights. Please see our responses to your questions below.\n\n**W1: Alternative scanning method.**\n\n**A:**: We justify the design of our method by providing new comparative studies for different position encoding schemes and alternative scanning methods. We first compare CCA with learnable position encoding. Due to resource limitations, we train all positional alignment approaches on 558k pre-training data and only 20k instruction data, including a new CCA model with same setup for fair comparison. We conduct evaluation on POPE and CHAIR benchmarks and the results are shown in tables below. The resulting models with learnable position encoding perform worse than our design.\n\n|**POPE (instruction-tune 20k)**|ran||pop||adv||\n|-|-|-|-|-|-|-|\n||acc|f1|acc|f1|acc|f1|\n|**MSCOCO**|||||||\n|learnable|85.93|85.83|83.80|84.02|77.73|79.28|\n|ours|**88.60**|**88.33**|**85.00**|**85.20**|**81.50**|**82.35**|\n|**GQA**|||||||\n|learnable|83.33|83.94|79.67|81.08|73.23|76.50|\n|ours|**85.63**|**85.97**|**79.83**|**81.37**|**75.13**|**77.98**|\n|**A-OKVQA**|||||||\n|learnable|83.47|84.01|80.50|81.67|72.27|75.80|\n|ours|**87.77**|**87.93**|**80.73**|**82.23**|**73.87**|**77.33**|\n\nWe also compare alternative scanning designs. We reverse the scanning order of original CCA and start from visual tokens at the center of image and end at periphery, illustrated in `Figure 2 (right)` of uploaded `pdf`. As kindly suggested in your question 3, we implemented diagonal scan and provide evaluation results for new scanning designs in a table below. Our original CCA scanning method demonstrated overall show better performance over other design choices. Based on these ablation studies, CCA is chosen as our final method for hallucination mitigation. \n\n|**POPE**|ran||pop||adv||\n|-|-|-|-|-|-|-|\n||acc|f1|acc|f1|acc|f1|\n|**MSCOCO**|||||||\n|CCA-r (new)|87.43|85.90|86.33|84.85|85.17|83.77|\n|diagonal-lora (new)|**88.10**|**86.72**|**87.17**|**85.83**|**85.70**|**84.46**|\n|CCA|88.03|86.65|86.87|85.54|85.67|84.42|\n|CCA-lora (new)|88.03|86.68|87.13|85.82|85.50|84.30|\n|**GQA**|||||||\n|CCA-r (new)|88.63|87.99|83.43|83.41|81.83|82.09|\n|diagonal-lora (new)|89.07|88.38|85.80|85.41|**82.70**|**82.77**|\n|CCA|**90.27**|**89.71**|**88.40**|**87.98**|82.30|82.74|\n|CCA-lora (new)|89.30|88.70|85.40|85.22|82.47|**82.77**|\n|**A-OKVQA**|||||||\n|CCA-r (new)|89.70|89.12|86.90|86.56|81.20|81.78|\n|diagonal-lora (new)|90.03|89.48|**87.93**|87.53|82.10|82.55|\n|CCA|88.40|87.68|86.47|85.91|**82.20**|82.37|\n|CCA-lora (new)|**90.33**|**89.88**|87.83|**87.59**|82.13|**82.70**|\n\n|**CHAIR**|512||||64||||\n|-|-|-|-|-|-|-|-|-|\n||c_s (↓)|c_i (↓)|rec|len|c_s (↓)|c_i (↓)|rec|len|\n|diagonal-lora (new)|49.0|13.6|79.6|93.1|**15.6**|5.2|64.9|52.6|\n|CCA-r (new)|50.0|18.1|**85.7**|96.4|18.0|5.5|66.1|54.7|\n|CCA|**43.0**|**11.5**|80.4|96.6|18.2|5.4|**66.7**|54.5|\n|CCA-lora (new)|45.0|12.4|80.8|93.3|17.2|**5.2**|65.4|52.7|\n\nFor presentation of concentric causal masking, please find new illustrations in `Figure 1` of uploaded `pdf`. \n\n**W2: Performance on CHAIR.**\n\n**A:** Thanks for pointing this out. We kindly remind that our trained model achieved the best results (lowest CHAIR scores) when applying CCA alone, as shown in the table below (summarized from manuscript Table 2). We point out that the LLaVA-RLHF model for benchmarking uses stronger Vicuna-13B as its language backbone and involves additional direct preference optimization stage in their training, whereas our model undergoes only two training stages (pre-training and supervised fine-tuning). Despite smaller 7B language backbone and less training compute, our model still outperforms the LLaVA-RLHF-13B model on most metrics.\n\n|CHAIR||512||||64||||\n|-|-|-|-|-|-|-|-|-|-|\n|||c_s (↓)|c_i (↓)|rec|len|c_s (↓)|c_i (↓)|rec|len|\n|LLaVA-RLHF-13B-v1.5-336|greedy|43.6|**10.5**|78.0|117.9|19.6|5.4|64.9|54.0|\n|OPERA|beam|46.8|13.4|79.6|93.2|17.8|5.9|64.3|53.0|\n|ours|greedy|**43.0**|11.5|**80.4**|96.6|18.2|5.4|**66.7**|54.5|\n|ours|beam|48.6|13.4|79.9|94.2|**16.0**|**5.3**|64.8|52.7|\n\nWe also point out that our method benefits POPE as well (manuscript Table 1), indicating good compatibility of our method on both open-ended generation and yes-no tasks. Moreover, we highlight that our method also benefits general perception tasks, where approaches that exclusively address object hallucination cannot always bring performance gain. Please refer to Table 5 in Appendix C.2, where we show that our trained model LLaVA-1.5-7B-CCA surpass LLaVA-1.5-7B over multiple LVLM benchmarks consistently.\n\n**Q1: Clarification on concentric causal masking.**\n\n**A:** In Figure 3 (d), we use different colours to highlight visual tokens with different positions. For a 2-D organization of visual tokens with shape of 6x6, our CCA leads to 3 positions in visual tokens. Please refer to our new illustrations in `Figure 1` of uploaded `pdf`, where query tokens, key tokens and masked tokens (tokens not involved in self-attention computation) are highlighted. Our CCA follows the same causal modelling rule in LLaMA Figure 1, where query tokens with larger position values attend to key tokens with smaller or equal position values. The difference is that we use a 2-D positional organization for visual tokens, which is a novel and effective attempt among existing LVLM hallucination studies. \n\n**Q2: Aggregated correct responses of concentric causal attention.**\n\n**A:** Thanks for mentioning this. We visualize aggregated correct responses in `Figure 5` of uploaded `pdf` with the proposed concentric causal attention. The resulting distribution is largely different from that from raster scan in Figure 2.a and reverse raster scan in Figure 2.b of our manuscript, showing 2-D and symmetrical distribution, which aligns with our concentric causal design. \n\n**Q3: Alternative position encodings and scanning.**\n\n**A:** Please refer to our response in **W1**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wd4JqhAXTd", "reviewer": "Reviewer_5R3r", "summary": "The paper attributes hallucinations in Large Vision-Language Models (LVLMs) to Rotary Positional Encoding (RoPE). It observes that LVLMs inherit a long-term decay issue from RoPE, where the inner-product of two tokens decays relative to their distance. This results in weaker visual-text interactions when the tokens are distant, leading to more frequent hallucinations if relevant visual tokens are far from the current generating token. To address this, the paper proposes a novel method called Concentric Causal Attention (CCA) to mitigate the effects of RoPE. Experiments show that CCA effectively reduces hallucinations and enhances the perception capability of LVLMs.", "questions": "1. In Figure 1, why are the attention weights higher at the beginning and the end? If this is related to the image content itself, then Figure 1 needs to show both the image and the textual query together to determine whether the attention is well distributed with respect to the query-object relationship. If the example is averaged over 3k POPE examples, this should be clearly stated.\n2. Is the method applicable to other widely used models like InstructBLIP [1] and Qwen-VL [2]? If these models do not use RoPE, the paper should mention this and discuss the implications.\n3. Does the CCA method end positional encoding at the center because objects are statistically more likely to be located at the center? Is this the reason for the higher attention at the beginning and the end in Figure 1(b)? Empirical evidence supporting this assumption should be provided.\n\n---\n\n[1] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning, NeurIPS 2023\n[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, Arxiv 2023", "limitations": "If the method is limited to LVLMs that use RoPE (e.g., LLaVA, MiniGPT), it is necessary to mention this as a limitation.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Touching the positional encoding aspect is quite fresh and particularly advantageous as it avoids the latency and multiple inference requirements seen in contrastive decoding methods such as VCD [1] and M3ID [2], which need two output probability distributions.\n2. Experimental results are quite strong.\n\n---\n\n[1] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding, CVPR 2024.  \n[2] Multi-Modal Hallucination Control by Visual Information Grounding, CVPR 2024.", "weaknesses": "1. The method is appealing as it addresses a different aspect than existing methods, but the writing can be improved in several areas. For example, the abstract should include short background information on RoPE and an explanation of the long-term decay problem, which will guide readers more friendly reading. Additionally, the captions for Figures 1 and 2 could be more concise and communicate more effectively to enhance readability. Figures 2 and 3 are difficult to understand and need clearer presentation.\n2. Since long-term decay is an inherent problem of RoPE, the paper should compare RoPE with other standard positional encodings (e.g., absolute, relative, learnable positional encodings) as well as some advanced positional encodings. This would provide a more comprehensive analysis of the issue.\n3. It is important to note that VCD [1] and OPERA [3] are training-free methods. Therefore, comparisons should also include more recent training-based methods.\n\n---\n\n[1] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding, CVPR 2024.   \n[2] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous reading and giving credit to our novelty and analysis of Rotary Position Encoding (RoPE) and LVLM hallucination. We appreciate you pointing out some additional references,  which we would include to make our research more complete. Please find our responses as follows.\n\n**W1: Clarification on RoPE.**\n\n**A:** Thanks for your kind advice. We will include a short background information on RoPE to indicate that, just as absolute and learnable position encoding, RoPE is also a type of position encoding, which is adopted by existing Large Language Models like LLaMA and inherited by most open-sourced LVLMs. We will also include a detailed guidance in Appendix on definition of RoPE and how it is involved in LLaMA architecture. For now we refer to [55] and lines 127-149 in our manuscript, where we present a mathematical form of RoPE and its long-term decay property for reference. For further clarification of Figure 2 in our manuscript, please find a new illustration in `Figure 4` of uploaded `pdf`. For Figure 3, please refer to `Figure 1 (right)` of uploaded `pdf` for a new illustration, where (a) to (f) corresponds to the first eight rows in Figure 3.d of our manuscript.\n\n**W2: Alternative position encoding.**\n\n**A:** Thanks for pointing this out. We kindly point out that RoPE is the default position encoding in LLaMA. Simply replacing RoPE with other position encodings is technically viable but deviates from LLaMA training scheme. We train a model with learnable position encoding and a model with relative position encoding [C] on this. Due to resource limitation, we train both models on 20k instruction data (instead of 665k) while train a new CCA model that follows the same setup for fair comparison. According to our results below, LVLM (learnable) performs much worse than LVLM (cca), while training of LVLM (relative) do not converge and not viable.\n\n|**POPE (instruction-tune 20k)**|ran||pop||adv||\n|-|-|-|-|-|-|-|\n||acc|f1|acc|f1|acc|f1|\n|**COCO**|||||||\n|learnable|85.93|85.83|83.80|84.02|77.73|79.28|\n|ours|**88.60**|**88.33**|**85.00**|**85.20**|**81.50**|**82.35**|\n|**GQA**|||||||\n|learnable|83.33|83.94|79.67|81.08|73.23|76.50|\n|ours|**85.63**|**85.97**|**79.83**|**81.37**|**75.13**|**77.98**|\n|**A-OKVQA**|||||||\n|learnable|83.47|84.01|80.50|81.67|72.27|75.80|\n|ours|**87.77**|**87.93**|**80.73**|**82.23**|**73.87**|**77.33**|\n\n**W3: Comparison with training-based methods.**\n\n**A:** We kindly remind that we compared our method with LLaVA-RLHF in Table 1 (for POPE) and Table 2 (for CHAIR) of our manuscript, a training method that mitigates object hallucination in LVLMs. We should point out that we compare our 7B model results against those from LLaVA-RLHF-13B model and we still stands out, indicating effectiveness of proposed method. As suggested, we compare CCA with a recent training method SeVa [B], which applies DPO training on LLaVA-1.5-7B. Overall our CCA model undergoes less training compute than SeVa, as SeVa applies another DPO training stage beyond LLaVA-1.5 pre-training and fine-tuning stage, while our training strictly follows LLaVA-1.5. POPE results are listed below,\n\n|**POPE**|ran||pop||adv||\n|-|-|-|-|-|-|-|\n||acc|f1|acc|f1|acc|f1|\n|**MSCOCO**|||||||\n|SeVa7B-MoCo|**89.43**|**88.88**|**87.23**|**86.88**|82.47|82.82|\n|ours|88.03|86.65|86.87|85.54|**85.67**|**84.42**|\n|**A-OKVQA**|||||||\n|SeVa7B-MoCo|89.96|**90.34**|84.33|85.70|75.57|79.35|\n|ours|**90.27**|89.71|**88.40**|**87.98**|**82.30**|**82.74**|\n|**GQA**|||||||\n|SeVa7B-MoCo|**89.27**|**89.73**|79.67|82.17|75.67|79.39|\n|ours|88.40|87.68|**86.47**|**85.91**|**82.20**|**82.37**|\n\nDespite less overall training compute, we highlight that our LLaVA-1.5-7B-CCA still outperforms SeVa-7B on 6 POPE evaluations. For more challenging adversarial evaluations , LLaVA-1.5-7B-CCA surpasses SeVa-7B by large margins consistently. We also compare our results with SeVa on LVLM benchmarks, where results of SeVa are taken directly from their paper. We show that our model surpasses SeVa-7B in most cases.\n\n|Model|SEED-all|SEED-img|SEED-vid|SQA|GQA|VizWiz|MMBench|MMStar|TextVQA|\n|-|-|-|-|-|-|-|-|-|-|\n|llava-1.5-7b|58.6|66.1|37.3|66.8|62.0|50.0|64.3|30.0|**58.2**|\n|vcd-llava-1.5-7b (new)|58.3|63.7|37.6|68.5|61.9|50.5|-|**34.6**|54.4|\n|seva-7b-diffu800 (new)|-|65.8|-|67.5|60.7|-|**65.6**|-|-|\n|ours|**61.7**|**67.1**|**41.0**|**69.8**|**63.1**|**57.6**|65.4|33.2|57.8|\n\n**Q1: Figure 1.**\n\nThank you for your question. We would like to clarify that higher aggregation values at the beginning and end of Figure 1.b (manuscript) is not related to image content as we averaged over 3k COCO images to get visualization results (as detailed in Appendix B.1). This may be attributed to removal of RoPE, which leads to out-of-training-distribution and breaks pre-trained LLaMA position encoding. \n\n**Q2: Applicability to Q-formers.**\n\nThanks for sharing this concern. Both InstructBLIP [14] and Qwen-VL [4] adopt RoPE in their language models. Nevertheless, our 2-D positional alignment strategy is designed for spatial-locality-preserved LVLMs [42,41,5], where the full image embedding from vision encoder is kept. Applying CCA to models like InstructBLIP and Qwen-VL is technically viable but not our design intentions. We will include this as a limitation of our method instead. \n\n**Q3: Empirical evidence on our concentric design.**\n\nYes. The concentric design is motivated by that more objects are statistically located at the centre. Please find a statistical evidence collected from COCO and GQA in `Figure 3` of uploaded `pdf`. Please refer to our response for **Q1** where we clarify results from Figure 1.b (manuscript).\n\n**L1: RoPE.**\n\nThanks for pointing this out. We admit that our method cannot apply to models with position encodings other than RoPE. We will include this in Limitation part. However, we think it is not a major drawback as most existing LVLMs use LLaMA as language backbone, where RoPE is applied as position encoding scheme.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate reviewers `9xbc` and `1NX2` for acknowledging clear motivation behind our work, and reviewers `5Msc` and `5R3r` for recognizing novelty of our study, along with thoughtful and kind suggestions for improving our paper. Please find new figures in attached `pdf`. New figures in our rebuttal texts are highlighted with `Figure x`. We also include new references here for all reviewers.

> [A] Towards VQA Models That Can Read.
>
> [B] Self-Supervised Visual Preference Alignment.
> 
> [C] Self-Attention with Relative Position Representations.

Meta-review: Summary:
This paper investigates how hallucinations in large vision-language models (LVLMs) are influenced by the RoPE long-term decay and its effect on vision information attenuation. The authors provide both visualizations and theoretical evidence showing that causal attention masks and RoPE embeddings are not well-suited for non-textual inputs like vision tokens. To address these issues, the paper proposes vision token re-organization and concentric causal attention masking, demonstrating improved performance in experiments.

Strengths:
The paper offers a strong explanation of the relationship between hallucinations and long-term decay in LVLMs, supported by both theoretical and empirical evidence.

All reviewers agree to accept the submission.
Decision: Accept (poster)