Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding
Authors: Hang Yin
Xiaomin He
Peiwen Yuan
Yiwei Li
Jiayi Shi
Wenxiao Fan
Shaoxiong Feng
Kan Li
Venue: NeurIPS 2025 poster ()
Keywords: data augmentation
multimodal language model
spatial understanding
Primary area: applications

Abstract: Existing vision-language models often suffer from spatial hallucinations, i.e., generating incorrect descriptions about the relative positions of objects in an image. We argue that this problem mainly stems from the asymmetric properties between images and text. To enrich the spatial understanding ability of vision-language models, we propose a simple, annotation-free, plug-and-play method named Stitch and Tell (abbreviated as SiTe), which injects structured spatial supervision into multimodal data. It constructs stitched image–text pairs by stitching images along a spatial axis and generating spatially-aware captions or question answer pairs based on the layout of stitched image, without relying on costly advanced models or human involvement. We evaluate SiTe across three architectures including LLaVA-v1.5-7B, LLaVA-Qwen2-1.5B and HALVA-7B, two training datasets, and thirteen benchmarks. Experiments show that SiTe improves spatial understanding tasks such as $\text{MME}_{\text{Position}}$ (+5.50\%) and Spatial-MM (+4.19\%), while maintaining or improving performance on general vision-language benchmarks. Our findings suggest that explicitly injecting spatially-aware structure into training data offers an effective way to mitigate spatial hallucinations and improve spatial understanding, while preserving general vision-language capabilities.

Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
Hang Yin', Xiaomin He', PeiWen Yuan', Yiwei Li', Jiayi Shi', Wenxiao Fan', Shaoxiong Feng', Kan Li
School of Computer Science, Beijing Institute of Technology
2 School of Software and Microelectronics, Peking University Xiaohongshu Inc [yh, pyh.peennnnnan liyiwei, shijjayyii wenxiaofan, likaa) edu. (2401210613)8stu, pku. edu cn, ( com
Abstract
Existing vision- lision-language models often suffer from spatial hallucinations, i.c., generating incorrect descriptions about the relative positions of objects in an image. We argue that this problem mainly stems from the asymmetric properties between images and text. To enrich the spatial understanding ability lision-language models, propose simple, annotation-free, blug-and-play method named Stitch and Tell (abbreviated as SiTe), which injects structured spatial supervision into multimodal data. It constructs stitched image- text pairs by stitching images along a spatial axis and generating spatially-aware captions or question answer pairs based on the layout of stitched image, without relying costly advanced models or human involvement. We evaluate SiTe across three architectures including LLaVA-v1.5-7B. LLaVA-Qwen2-Ia. and HALVA-7B. two training datasets, and thirteen benchmarks. Experiments show that SiTe improves spatial understanding tasks such as MMEPosition (+5.50%) and Spatial-MM (+4.19%) while maintaining or improving performance on general vision-languag benchmarks. Our findings suggest that explicitly injecting spatially-awas awwree structure into training data offers an effective way to mitigate spatial hallucinations and improve spatial understanding, while preserving general vision- languag capabilities.
Introduction
Spatial understanding, the ability to comprehend and interpret the relationships between objects in space, is essential for tasks such as visual question answering, navigation, and embodied AI [| 40). However, most existing vision-language models still struggle to understand and reason about spatial relationships 19 34 39], which leads to spatial hallucination problems.
We argue that the spatial hallucination problem is primarily caused by the implicit modality gap, that is, while images contain rich multidimensional spatial features, their paired captions tend to be relatively sparse. We investigated existing large-scale multi-modal datasets, such as Conceptual Captions (28), COCO [18, VQA M., and SBU Captions 24), and observed only a small fraction of samples contain explicit spatial information (see Table D As shown in Figure I the spatially-aware data means the samples whose captions include clear spatial information (c.g.. "to the left of",, "in front of"") During training, the model aligns visual content with limited linguistic descriptions, which may constrain its ability to capture latent spatial structures from the image alone.
This further motivates the need to introduce spatial information directly into the text. However, collecting spatial annotations through nowddsoureing is both difficult and expensive. often demands carefully designed annotation interface and large-scale annotation efforts involving skilled
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Spatially-agnottt text A stop sign stands quiet street, woman walks across the crosswalk.
Table 1: Proportion of spatially-aware data in common datasets.
STOP
Multimodal Datasets
Ratio
Conceptual captions 28
0.0168
Spatially-awe text:
blip laion oc _______5_KKKK_
558K
20
0.0201
left the stop sign, a
VQA v2 [
0.0288
woman is walking across the street.
COCO COCO_2017 2017
0.0511
SBU_Captions
23
0.0582
Figure I: The difference between spatially-agnostic and spatially-aware text. spatially-aware text has explicit location cues that clarify object positions.
15
0.0611
Flickr30K 43
0.0732
VSR 19
0.1898
annotators. One might consider leveraging data augmentation to construct spatial understanding data. However, traditional methods not design for spatial understanding. For instance, cropping, dithering. rotation, and random erasing may break the alignment between images and text, introduce spatial noise, large models to synthesize spatially-awar data through caption rewriting or image editing (35] Although model-generated augmentation enrich spatial supervision, it typically involves high computational cost and complicated processing. Given the massive scale of pretraining data, such methods are difficult to apply in practice.
distort main semantic consistency [6] [4] 29]. Recent work has explored using
Raw
Image-Spatie Stitching
Text Spatial Telling
"RAVN
Ti the raven legacy of master thief
RAVEN
Ty a porch of reckytop, o bedroom cabin nantal located in pigeon country
Left? Right?
the roven legocy of moster thief
Spatiol information injection
Honizontalt Stitch/..
Left side diop lys thh rovan logecy of master tiver While the right side shows porch reckytop,
Taitck
ROWEN
located pigson country. Right imoge dep the raven legccy of noster thief) and the left imoge which shows 8 porch of rocky a bedroom capin rental lscated in pigcon country.
porch rockytop, bedroom cabin rental located in pigeon country
From observer's iy point viow, (theff
located to of (parch]?",
Left!
Right!
(Taines. Tame)
Figure 2: An Overview of Stitch and Tell
In this work, propose simple yet effective multi-modal data augmentation method, Stitch and Tell (SiTe), which injects spatial information into image- text pairs without relying on large-scale generation models incurring heavy computational cost. As shown in Figure 2 SiTe consists of two main steps: IMAGE-SPATIE STITCHING and TEXT-SPATIAL TELLING.
In the IMAGE-SPATIAL STITCHING step, we combine two images along spatial axis (e.g., horizontal or vertical) to create a new image with an explicit spatial layout. This stitched structure naturally introduces spatial relations between regions.
In the TEXT-SPATIAL TELLING step, we generate spatially- awwaee textual annotations based on the original captions. First, we construct structured captions by placing the two original texts into spatial templates, such "The right part shows T2, while the left part displays Ti." These templates make spatial relations explicit and provide clearer supervision, helping the model associate language with visual layout. Second, we extract object nouns from each caption and generate spatial question answer (QA) pairs. For example, given "a cat" from Ti and "a car" from T2, we create question like "From the observer's perspective, is the cat on the left the car?" The answer can automatically determined based on the stitched layout, requiring no manual labeling. These QA examples offer an

---

additional form of weak spatial supervision and can be directly used in instruction tuning. Compared to image-based object detection or grounding, extracting object from text more efficient and reliable. It reduces the noise and computational overhead introduced by vision-level processing, making our method lightweight and scalable. The image-text pairs generated by SiTe introduce explicit spatial supervision through structured layout and spatially- text, without requiring manual annotations or model-generated rewrites. This helps bridge the gap between sparse spatial expressions in text and the richer spatial structure present in images, improving cross-modal alignment.
SiTe introduces spatial structure into multi-modal training data through simple image stitching and spatially-aware caption generation, without relying on large-scale generation models or human annotations. Each stitched sample has two image- text pairs, allowing the model to process more content per step and reducing training time by over 20% in our SiTe default setting. SiTe is easy to integrate into existing pipelines. can be applied during both pretraining and supervised fine-tuning without modifying the model architecture.
We evaluate SiTe across three model architectures (LLaVA-vI.5-1) 20), LLaVA-Qwen2-1.3 [I]
and HALVA (27)). two training dataset (558K (20) and Flickr30K), and thirteen popular bench- marks, four for spatial understanding, four for general vision-language tasks and five for more real-world benchmarks. On LLaVA-vI.5-7B. SiTT improves spatial understanding benchmarks such as MMEPostiion (+5.50%) and Spatial-MM (+2.13%). while also yielding gains on general tasks like COCO-QA (+1.02%) and MMBench (+0.93%). For Qwen2-1.55 1.5B, SiTe improves MMBench by +5.33% and MM-Vetspe by +4.38% in the fine-tuning stage, and achieves +10.42% on MMEpss and +5.01 % on MMBench during pretraining. These results demonstrate that SiTe provides an effective form of weak spatial supervision that enhances spatial reasoning while maintaining competitive performance on general benchmarks, across both large and small model settings.
2 Related Works
Multimodal Data Augmentation. To improve the generalization of vision-language models, recent
work explores multimodal data augmentation strategies [13 26). MixGen [12] combines image
interpolation with caption concatenation, but assumes semantic compatibility across samples, which may yield implausible pairs. Other approaches rely on generative models, such as StableLLaVa [66] for diffusion- based synthesis and ALIA [9] for language- these methods into input mixing, caption synthesis, and adversarial perturbation. However, most require supervision, handcrafted rules, or heavy computation, limiting scalability. In contrast, we propose structured, weakly supervised augmentation method based spatial compositionality. It introduces explicit spatial grounding without labels or model-generated text, and integrates efficiently into standard training pipelines.
editing. Sapkota et al. [26] categorize
Spatial Understanding Task Spatial understanding is a fundamental capability for intelligent agents to recognize and reason about the relative positions and relationships between objects. It plays a central role in real-world applications such as robotics, embodied AI, and autonomous
emphasize that spatial reasoning is critical for scene understanding. navigation, and visual question answering, all of which require accurate perception of object positions and layouts. In robotics and autonomous systems, it enables agents to make informed decisions in dynamic environments [46, 2 g 22). In the vision- lision-language domain, spatial information essential for grounding language in visual content [33 25). To enhance spatial reasoning, recent efforts incorporate explicit spatial features such as coordinate embeddings 4 and leverage additional modalities like depth and 3D information [ 10 The increasing attention to this area reflects both the challenges it poses and its importance to general intelligence, making it an active field with strong potential for future progress.
driving [11] 21 3 Recent studies [30 31
45
40 42
Stitch and Tell
In this section, we present Stitch and Tell, structured multimodal data augmentation method that injects spatial knowledge by stitching images and telling their spatial layout information through structured text. We begin by introducing how images are spatially stitched to form structured visual information. We then describe how spatial relations are explicitly injected into both captions and question answer formats, allowing the model to learn spatial reasoning from weakly structured but
3

---

naturally aligned supervision. Finally, we discuss how SiTe can be effectively integrated into the model's training process.
IMAGE-SPATIAL STITCHING: Given two image-
caption pairs (7,, Ti) and (I2, T2) from the dataset, we first construct stitched image by spa- tially stitching the two input images along spe- cific axis. Concretely, we create a blank canvas based on the larger height (for horizontal stitching)
Algorithm Image Stitch Algorithm
function STITCH(/1, 12, mode)
2:
(w1, h1) ) GetSize(
(w2, h2) f- GetSize (2)
or width (for vertical stitching) of the input images,
4:
if made = horizontal then
5:
H max(A1,
and paste 1 and Ig onto the canvas following a layout mode. For example, in the horizontal set- ting, we produce left-right stitched image via TLR
6:
W wi u22
Luumas New.Imgee H)
8:
Paste 1, at 0) onto camvss
Stitch 1 @horizomaa 12. This construction intro-
9:
Paste 12 at (w0, 0) onto Lanaa
duces an explicit spatial information compaerd 10:
else if mode vertical then
the original images, helping the model acquire a grounded sense spatial. The pseudocode of im- age stitching process is illustrated in Algorithm
11:
W max(wi, w2)
12:
<- h1 h2 Taams <- NewImage(
13:
14:
Paste 11 at (0, 0) onto Leunns
We design two image pairing strategies for con- structing composite samples in the SiTe frame-
15:
Paste (0, ht) onto Lanmss
16:
else
work: (1) SiTerand randomly selects two images
17:
return Error: Invalid mode
from the dataset for horizontal concatenation,
18:
end if
without considering their geometric proportions. This introduces diverse visual combinations but may lead to uneven scaling or excessive blank ar- eas in the merged image. (2) SiTematio first filters vertically dominant images with a height-to-width ratio greater than 1.2, then groups them into buckets according to similar aspect ratios, and pairs images within each bucket. This ensures that the two halves of the composite image have comparable geometric structures, thereby improving spatial balance and reducing blank redundant regions. Compared with the random pairing baseline, SiTe,atio increases the proportion of effective visual tokens and yields higher information density in the visual encoder's representation.
19:
return Teaess
20: end function
TEXT-SPATIAL TELLING. After performing image-spatial stitching, such as horizontally combining two images, obtain new image Smich with an explicit spatial layout.
LR
Tell the Caption,The semantic content from the original captions Tj and T/ is naturally aligned with the left and right regions of the stitched image. Based on the stitching mode (e.g., left-right tow-down), we select spa- tial template from repository and stitch a struc- tured caption Tsnch by inserting Tj and Tz into the corresponding placeholders. For ex- ample, the template "The left part showy Ti, and the right part displays T2." explicitly in- jects spatial information through the underlined spatial phrases. These spatial information help the model associate textual semantics with the visual layout more effectively. This process does not require any additional annotation, and re- tains the original semantics while explicitly encoding spatial relationships, encouraging the model better align visual content with spatial language.
kittenn on the floor wearing small hot.
puppy sits on the gross.
Dotten, floor, hat)
No -ppppr, grass)
(kimman) appear (pusps)
"Yes
Figure 3: A construction process of spatial visual question answering data
Tell the Question and Answer. Guided by the SiTe method, we further extend our method to generate spatial question- answer pairs. As shown in Figure 3 for each stitched sample, we extract noun lists from the original captions: Nt = [oL.1.... OL. and NR = [OR,1...., RR,]. Suich To reduce ambiguity, we remove
corresponding to the left and right regions of the image 7LR

---

overlapping nouns shared by both sides. We then sample entity pairs (oL, from the disjoint noun sets and generate questions such as "Is the or, on the left of the on²?"
Since the spatial position of each entity is predetermined by construction, the answer to such questions can be automatically inferred based on the object's region (e.g., oz. Nr implies a "Yes" answer). This provides a lightweight way to generate weakly supervised spatial reasoning signals without the need for manual annotation. Compared object detection or image-level classification, extracting entities from captions is significantly more efficient and reliable. It avoids visual ambiguity and reduces computational cost. These question-answenn pairs provide the model with a lightweight but effective signal for learning spatial understanding.
The data generated by SiTe can be effectively applied across multiple stages of multimodal training. The structured captions Tsich provide spatially grounded supervision during pretraining, facilitating the alignment between visual representations and spatial language without additional annotation. The constructed spatial question-answer pairs are suited for instruction tuning, where they serve as explicit prompts to strengthen the model's spatial understanding abilities. Furthermore, by modifying spatial expressions in 7 Stithh (e.g., exchanging "left" and "right"), we can generate hard negative samples that maintain the global semanties while introducing localized spatial inconsistencies. These samples can be used in contrastive learning settings to improve the model's sensitivity to spatial contradictions. Overall, SiTe offers a unified and extensible method for spatial data augmentation, supporting both generative and discriminative learning objectives with minimal supervision.
4 Experiment Setup
4.1 Models.
We conduct experiments using three representative vision-language models: -LaVA-v1.5-74 (0] (referred to as LLaVA), LLaVA-Qwen2-1.5BB [41] (Qwen2), and HALVA-7B (27] (HALVA). LLaVA built on Vicuna-7B, while Qwen2 uses Qwen2-1.3 as the language backbone. HALVA adopts a contrastive learning framework by aligning correct and hallucinated phrases at the token level for fine-grained supervision. We evaluate LLaVA and Qwen2 both pretraining and fine-tuning stages, and apply HALVA only in the fine-tuning stage.
4.2 Training Sets and Augmentation Setup.
The training process of LLaVA consists of two stages: the pretraining stage and the supervised fine-tuning stage. We describe the setup for each stage separately.
Pretraining Stage. To evaluate the effectiveness of SiTe across different training sets, we exper- iment with two image-caption datasets: lhap_laaon_cc.oo sc___bbu_aaa (558X) and Flickr30K. As shown in Table I these datasets differ in the proportion of spatially informative samples. with Flickr30K containing a higher density of such spatial descriptions compared to 558K. We apply both horizontal and vertical stitching in a 1:1 ratio. By default, the stitched data make up one-third of the total set. To avoid duplicate supervision, mmage-caption pairs used for stitching are removed from the raw set, ensuring each image appears only once. As result, the final number of training samples is slightly smaller than 558K. We design 35 templates for horizontal stitching and 29 templates for veritcal stitching. For each sample, template is randomly selected from the corresponding set based on the stitching mode. This diversity in spatial phrasing encourages the model to learn spatial relations in more flexible and robust manner, rather than relying on fixed linguistic patterns. To further compare SiTe- -uge-nugentedd data with existing spatial supervision data, we construct a pretraining variant by substituting part of the original mapeccaptiion data with equal-sized spatially -fouused samples from the Visual Spatial Reasoning (VSR) dataset [9). In default setting, the number of VSR data is 5K. Additionally, Crop. For Rotate, images are randomly rotated between 0 and 360 with captions unchanged. For Crop. 80%-100% of the image randomly retained and cropped from a random region. Further implementation details, including the full list of spatial templates used for horizontal and vertical stitching, as well as examples of the stitched image-text pairs, are provided in Appendix B
compare SiTe with two standard augmentation baselines: Rotate and
Supervised Pine-Tuning Stage. We apply the SiTe method to augment the 1lava_ v1_5 5_mix665k dataset. Noun phrases are extracted from original sentences using Qwen2. 5-72B- 10--72B-Inns Instruct with the
5

---

"Extract the concrete, visible physical objects or entities described in this sentence. Return a conma-separated list. Ignore abstract terms like 'type', 'color', 'time', or actions." " Based on the extracted entities, we construct spatial anestion-answer pairs to form a new instruction-tuning dataset. We evaluate the effect of adding IK and 5K such samples to both L.LaVA and Qwen2 models.
prompt:
We further explore the way to construct spatial negative data in contrastive learning by SiTe, For each stitched caption, we generate negative example by swapping the spatial expressions. For example, given a positive sample such as "The bottom image contains 7',, and the top image shows 72," we construct a corresponding negative sample as "The top image contains T1, and the bottom image shows T2." The SiTe method simplifies this process by providing explicit spatial layouts, making it easy to create contrastive pairs based on spatial inconsistencies.
For the supervised fine-tuning stage, we define more than 20 question templates to generate spatial reasoning prompts. To reduce ambiguity, most templates explicitly incorporate the camera's per- spective, using phrases such as "from the camera's 's point of view, is left_obj located to the left of right_obj?" clarify directional references. See Appendix for more details about QA templates.
SiTe provides a simple and controllable way to construct such contrastive pairs. By modifying spatial expressions in structured captions (e.g., swapping "left" and "right"), we can easily generate hallucinated variants while preserving the global semantics. In default setting for HALVAsHe use 20K data, and the HALVAgaseine use 21.5K data.
4.3 Training Setup.
For the pretraining stage, we train L.5VA-v1.5-7B and ILLVA-Qwen2-1..5 using a batch size of 16 and 64 per GPU, respectively, on 8 L.20Z GPUs. All pretraining experiments conducted for 1 epoch following the original LLaVA setup. For the supervised fine-tuning stage, we use the same hardware setup and follow the original LLaVA configuration, training both models for epoch. The batch sizes are set to 4, 16 and 128 per GPU for HALVA, LLaVA-v1.5-7D and LLaVA-Qwen2-1a. respectively. We adopt the same learning rate and weight decay as in the original model settings. In all ablation experiments, the batch size and training schedule also remain consistent, and only the ratio of stitched data is varied. For each setting, we run five times and report the average results.
4.4 Evaluation Setup.
We assess models in two key areas: spatial understanding and general vision-language capabilities. For spatial understanding, we use four benchmarks: COCO-QAspa (subset of COCO-QA focusing on spatial questions), Spatial-MM (multiple-choice benchmark testing spatial relations between objects), MME-Position (MME subset with spatially-aware questions answered in a single word/phrase), and MMsgat (MM-W) Vet subset evaluating complex spatial reasoning). To examine whether spatial supervision affects general multimodal performance, we evaluate on COCO-QA (a multimodal dataset for basic visual understanding). VQA-v2 (diverse couman-annotaan
QA pairs from MS- IS-COOC for robust visual reasoning), MMBench (multi-choice benchmark covering 20 vision-language ability dimensions), and MM-Vet (open-ended and multi-choice tasks testing integrated reasoning and knowledge). To verify the model's ' generalization ability in spatial understanding, we conducted evaluations CV-Bench [32] and RealWorldQA (38), both of which include questions involving 3D spatial reasoning. In addition, to explore whether this approach can also enhance model performance on high-resolution data, we carried experiments on two high-resolution datasets, HR- Bench 8K (36 and V-Star All evaluations are conducted in zero-shot setting on target benchmarks.
5 Main Results
In this section, we present the performance of SiTe and several baseline methods on both spatial understanding and general vision-language benchmarks. Notably, the pretraining stage, the total number of training samples is kept consistent across the Baseline, Rotate, Crop, and VSR settings. For SiTe, since each stitched sample is formed by combining two image-caption pairs, the total number of training examples is reduced accordingly to ensure the model sees the same number of unique images. This means that no image-caption caption pair appears in both the stitched and raw data. This adjustment ensures a more fair comparison across different stitching ratios.

---

Table 2: Performance comparison on spatial understanding and general vision-language benchmarks with 1 or + values showing improvements or declines relative to each corresponding baseline. In the supervised line-tunin stage, the superscript (e.g., LLa indicates that 1K spatially- aware samples are added per stitching direction. The bottom-right corner of each model denotes the data augmentation method used. Each variant is compared to its corresponding baseline with the same backbone or data setting. *Results using their official checkpoint.
Model
Spatial Understanding
General Vinion-Language
CoCo COCO-QAguu (%6) SpatialMMM(3) MM(33)
MMEEp
MM Vetogn
COCO QA (QN22(%) MMB/S) MM Net
Pretesining Shoge
LLAVAnadine
67.72
42.02
127.83
26.52
70-52
60.48
31.11
LLaYAggue
68.09
43.01
128.99
29.40
71.36
60.60
74.86
32.20
LLeVACop
67.82
42.69
127.78
70.93
60.37
31.43
LLLYAAyR
67.85
42.05
121.67
29.42
70.50
60.57
73.88
31.47
LLaVAsTTerand
61.75
43.78
133.33
28.4371991
71.54"100
60.5310007
76.4379995
79.7
L.LVAsTeratle
70,06
44.1% 44.15
132.80 1-99
28.68
71.19 79.6
60,57
73.64
$2.27
LLaaA
44.00
129.00
71.75
72.33
LLAVA
68.03
44.81
129.44
25.86
71.12
59.42
72,36
29.60
LLLVA
09.34
42.09
131.11
28.73
72.12
60.20
72.68
29.87
LLaVAygE
68.26
42.70
128.75
26.18
71.33
59.94
72.87
30,45
LLaVA Bikr
71442
44.97 19.94
100 11:00
26.20" 79022
73.74 71.94
59.73110110
72.88 10.55
20.55 Trons
LLaVA Bick
SiTc-raiie
71,51 12.55
44,31 T0.29
131,50
28.00
13.62
73.89
59.94470.33
59.544
70.31
71.66-49.67
71
30.97 11.43
Qwen2guzlie
62.25
40.85
60.75
20.72
64.72
$3.66
61.67
22.48
Qwen2goone
60.73
40.78
60.50
19.62
60.22
52.77
66.61
20.97
Cien
62.06
40.68
61.25
20.75
64.74
53.28
67.45
22.52
Owen2vsR
57.18
41.46
58.30
22.40
60.22
54.64
66.58
22.95
rae
62.57 10.33
11:25
63.0
72.9
65.26 70.54
11.18
28.68771
15.050 01
10-33
QuanZaTi-ratle
62,52 19.37
41.00 19.15
68.00
21.00
65.10
54,23
67.57
22.80
70.80 79.92
Qwen2fick Rick
38.90
51.15
10:95
47.90
42.38
10.50
Owenzfiake Rogie
40.08
30.10
$8.33
9.50
42.45
79.04
47.86
9.20
Qwen
42.37
39.37
64.25
8.30
44.61
40.41
46,43
10.00
Qwen Slcke
30.90
67.25
10.15
48.00
40.95
48.71
9.60
Quena Derandd
47.71 1051
39.35 39.357045
61.67
61.6771042
12:47 71.50
48.68 10.78
42.95
42.9570557
10.57
51.33
51.337043
70.45
70.30
Quent To-ratle
49.04 11.04
40.04
56.00 14.75
13,10
47.38 10.51
43.91
11.55
51,31
10.60
79.0
Sxpervind Fine-tuning Sioge
LLaVA Sterand
11.0
43.16 1.14
128.7019.8
27.06 70.54
71,24
61.1710.6
74.00
70.20
LL IK
SiTeraii
68.35
46,96 1494
13.6
2917
71.32
60,22
73,374013
31.54
M.aVA Niezznd
SK
14.7
11443
28
70.96 70.44
74.53
50.69
LL
STTeratii
68.371005
48.5876.56
141.00
31.08 74.57
70.9270.00
60382
73.36
12.15
Quen
Tet-raad
58.534375
65.33
23.10 23.10
61.77-795
$1.58
66.37
23,93
Quen
59,664259
41.31
62.751200
24.3
62.59/213
1..3
65.75
1408
22.070032
Quann Boorand
19.9513.10
41.22
61,671992
23.20
63.00
14.75
66.43
66,4314476
23,93
Quena
60.2242.88
41.56
64.50
25.10
62886-136
S4.971.11
67.00
75.33
24.27
HALVAnuutte.
63.16
43.07
135.00
25.70
67.32
61.67
72,44
30.00
HALYASTT
6..77
44.151
123 221333- 1167
26. 10.10 10700
68.56
61.00
71,544090
30.s0
5.1 Evaluation on Spatial Understanding Benchmarks
We first evaluate the impact of SiTe on spatial understanding. The results are provided in Table 2 In pretraining stage, experiments are conducted under the pretraining setting using two datasets (558K and Flickr30K) and two backbones: I.LVVA-v1.5.. 1-577- and LLLaVA-Qwen2-1.55 We compare SiTe against baseline training, traditional augmentations (Rotate, Crop), and training data mixed with VSR data.
Effect of A'ge-Augeet Pretraining. SiTe-augmented method consistently improves spatial understanding performance across all benchmarks and model backbones. This gain can be attributed to its explicit introduction of structured spatial knowledge during pretraining. By stitching two images with corresponding spatial information (e.g., "left to". "top of"), the model learns to associate language not only with object content, but also with relative layout.
The V Rotate model shows moderate improvement on the 558K dataset. This may be because spatial ex expressions are relatively sparse 558K, so rotating the image does not heavily conflict with the caption. In some cases, it may even help the model become more robust by learning to generalize
7

---

across different viewpoints. However, when using Flickr30K as the training set, where captions contain richer spatial descriptions, LL flickr
Rotale performs worse than LLaVA llickr on several spatial benchmarks. For instance, on COCO-QA Spu accuracy drops by 0.93%. This suggests that rotation may introduce misalignment between spatial phrases in the caption and the actual image layout, potentially resulting in implicit negative supervision. And the performance of the Crop baseline is less stable. Random cropping may inadvertently remove key objects or semantic regions, leading to weakened image-text alignment. Although the retained area is still between 0.8 to I, the risk of disrupting mismatch grounding remains potentilly introducing merrors where the caption does not accurately reflects the image content.
Among SiTe variants. SiTe-rand corresponds to the original random pairing strategy, while SiTe-ratio adopts a ratio-based image pairing scheme that aligns images by aspect ratio before stitching. Compared with SiTe-rand, SiTc-ratio achieves further gains on most spatial benchmarks, indicating that more efficient image composition improves the performance of spatial supervision. The random variant still provides strong overall enhancement across models and datasets, showing that SiTe is robust even without additional pairing constraints.
Flickr30K contains higher proportion of spatially- aware data, which helps allows LLa flickr learn both spatial relations within each image and layout patterns from stitched pairs, and perform better than LLa VALae 68.75% to 71.42%). This allows the model to better understand complex spatial structures, even with less data in pretraining, and achieve strong overall performance.
Effect of SiTe-Augmented Supervised Fine- Fine-Tuning. SiTe continues to deliver strong results during the supervised fine-tuning stage. By adding IK and 5K spatially-aware samples per stitching mode to the instruction tuning set, we observe consistent improvements across most spatial benchmarks. For example, after adding 5K horizontal and 5K vertical QA samples to ALaVA-v1.5-77 in fine-tuning stage, the accuracy on Spatial-MM increases from 42.97% to 46.21% %. outperforming the baseline by 4.19%. These gains come from our data construction strategy, which extracts noun entities from stitched regions and generates spatial QA instructions from the camera's 's perspective. This enables the model to learn spatial reasoning patterns from naturally aligned weak supervision.
The au--meteeddd method can also be used to construct contrastive examples. We apply spatially stitched text HALVA by replacing only spatial information (e.g.. "left" "right") while keeping the other semantics information unchanged. In contrast, the HALVA method modifies objects, relations, and attributes. Despite this simpler setup, HALVASITe still outperforms the HALVABuselne Baseline most benchmarks.
We observe slight performance drop for Qwen2srre on COCO-QA. This suggests potential unalign- ment when fine-tuning mee-tunnn small-capacity models on highly structured spatial QA data. Specifically, SiTe provides binary Yes/No questions focused spatial understanding, which differ in format and answer space from COCO-QA's diverse and open-ended questions. As a result, the model may over-adapt to the binary QA style, leading to reduced flexibility when answering questions that require generative reasoning or retrieval of specific object names. This effect is more pronounced in smaller models like Qwen2-1.558 which are more sensitive supervision bias and have limited generalization capacity.
5.2 Evaluation on General Vision-Languaae Benchmarks
We further evaluate whether SiTe compromises general vision-language capabilities while improving spatial understanding. Results are summarized in Table 2 For models based on L.aVA-vv.5-7BB, SiTe consistently maintains or improves general performance. Notably, SiTe-augmented in LLaVA maintains consistent improvements on general vision-language benchmarks during the pretraining stage. This suggests that introducing spatial understanding at this stage enhances the model's ability to generalize to diito--distribution (OOD) scenarios. In the supervised fine-tuning stage stage, SiTe still shows mostly positive gains, though we observe slight drops on VQA-v2 and MM- Vet. This may be due to the nature of instruction tuning, where large number of spatial-focused examples shift the model's 's preference, potentially affecting its performance on general tasks. For Qwen2, which has relatively limited capacity, SiTe also brings clear benefits in pretraining. In particular, Qwen2 improves MMBench by 5.01% %. However, during supervised fine-tuning stage, Qwen2 shows a slight decrease on COCO-QA, indicating that instruction tuning tends to reinforce domain-specifie behavior. For smaller models, this may reduce the ability to follow diverse instructions, leading to weaker
8

---

generalization. These results further highlight the importance of spatial supervision during pretraining improving downstream robustness.
We further conducted experiments on both high-resolution and 3D spatial benchmarks to evaluate the generalization capability of approach. Specifically, tested models enhanced with SiTe on HR-Bench 8K and V-Star, where they consistently achieved higher scores than the baselines, demonstrating improved perception fine-grained visual details.
To assess spatial reasoning in more realistic and complex settings, we also evaluated the models on CV-Bench and RealWorldQA, which include diverse questions involving 3D spatial relationships. The results show that injecting structured spatial supervision through SiTe significantly improves performance out-of-distribution spatial reasoning tasks and generalizes well to challenging real-world scenarios. We hypothesize that these gains arise from the model's enhanced ability to represent fundamental directional relations, which in turn supports broader spatial understanding across different dimensions.
Table 3: Performance comparison across real-world and high resolution benchmarks.
Model
CV-Bench 2D (S)
CV-Bench 3D (%)
RealworidQA (%)
HRBiench-K (%)
V-Star (%)
LLaVAgucine
32.56
33.43
55,12
30.90
48.54
LLa VAstleodd
$1.28 71:73
35.5
70:09
71.00
89 1161
LLa MagAeeoemii
55.38 13.28 13.02
36,5313.00
55.2610114
23,721282
50.13
LLAVA
52.87
28.39
34.51
33.52
49.32
Bickr
71.067.19
76627
75.2870077
24.537101
47.28-2.04
LLaVA Stil-muli
$5.07
39.80
56.25
33.94
50.79
Quen
46.77
42.63
52.28
32.25
37,83
Qwenlgre-rad
48.16 71:16
50.361275
56.4473166
32.671044
38 38.92 11.00
OwenngTTe-ni
47.4770.70
50,4012.7
52,947066
32,9470.00
38.09
Qwent
42.59
51.57
43.20
32.63
36.13
Owendding flickr
73.64
51.6770.10 1o
42.7540045
73.3
70.99
Owenz
44.44
51.5049.07
49.74
34.33
36.29
We also calculate the computational time efficiency improvement brought about by pre-training training using SiTe method. Taking default as an example, due to the decrease in the number of data strips, the time
epoch in the pre-training stage is 77.4% of the baseline.
In summary, Stitch and Tell effectively enhances spatial supervision by increasing data density through structured augmentation, without sacrificing the original knowledge. This approach leads to stronger spatial understanding and remains competitive on general vision-language benchmarks, demonstrating its broad applicability.
5.3 Qualitative Analysis
Is lea cup on of cake from camera's perspective? A. Yes B. No
Where is the AI chip in relation to the hand? A. Front B. Behind C. Above Under
(u)
LLaVA:A
SiTe:
(b)
LLaVA:
Is there a backpack on laptop from camcra's perspective?
there table lamp on the left of the TV from camera's perspective? A. Yes B. No
A. Yos B. No
(c)
LLaVA:A
SiTe:
LLaVA: B
SITe: A
Figure 4: Qualitative comparisons between LLaVA and SiTe. SiTe can better distribute attention between objects and judge their spatial relationships. We visualize attention differences MI.LM-Knoww [14, which highlights image regions where the model assigns more attention under task-specific prompts compared to general prompts. A qualitative

---

comparison between LLaVA-v1.5-7B and SiTe is shown in F In Figure (a), when asked about the relation between the AI chip and the hand, LLaVA mainly attends to the hand while neglecting the chip, resulting in an incorrect answer. In contrast, SiTe correctly focuses on both entities. In Figure(3 (b), where the question involves the left side of the cake, SiTe allocates more attention the region between the cup and the cake, effectively capturing their spatial arrangement. Similarly, in Figure (c) and (d), SiTe exhibits more precise attention distribution across the referenced objects, which contributes to more accurate spatial understanding.
5.4 Ablations
Recalling the SiTe method in the pretraining stage, we define a stitching ratio parameter À to measure the proportion between stitched and raw samples in the final training set. Let 7 denote the number of original image-captions pairs, and N be the number of stitched samples constructed for each stitching mode (e.g., horizontal or vertical). Since each stitched sample is formed by consuming two raw samples, the total number of stitched samples is 2.V, and the remaining raw examples become T 4N, assuming two stitching modes are used. Let Ssith and DRW denote the sets of stitched and retained raw examples, respectively. The final ratio À is defined as:
[Panxll 2NN
[aom T-4N-
Table 4: Settings of SiTe-rand as variants of deTedtf used for the ablation study during pre- training on the 558K dataset. Each variant adopts different ratio X, representing the proportion of stitched samples to the remaining raw samples in the training set, and the total number of training instances varies accordingly.
We present an ablation study using the 558K dataset for pretraining. Specifically, we intro- duce seven SeTi variants with different stitching ratios, as summarized in Table 5 We observe that when the stitching ratio À is around 1/3, the model achieves overall strong performance on spa- tial understanding benchmarks while maintaining competitive results on general vision-language tasks. This setting provides a favorable trade-off between spatial supervision and overall data diver- sity. A similar trend is observed when using the Flickr30K dataset for training, and the more detail ablation results and analysis are included the Appendix C
Setting
Images Total data size)
SiTeataant
458K
1 3.58
(a) N=1K
556K
1: 277.0
N=5K
548K
53.8
N=10K
538K
25.9
(d) N 100K
358K
1: 0.79
(e) 139K
280K
1: 0.01
Table 5: Performance on spatial understanding benchmarks (left) and general vision- vision-language benchmarks (right) under different spatial data augmentation settings.
Setting
Spatial Understanding
General Vision-Language
COCO-QA Squniil
Spatial
Spatial-MM
MM MMssiiio
MM- Vetspa
COCO-QA
VQA,2
MMB
MM- Vet
Si Tedeaut
68.75
43.78
133.33
28.43
74.43
60.53
74.43
31.40
(a)
69.23
42.90
126.3
27.54
74.05
60.50
74.05
30.58
(b)
67.58
42.42
132.67
27.59
74.11
60.35
74.11
31.34
(c)
68.13
42.83
128.67
27.50
74.04
60.40
74.04
31.45
68,83
43.10
132.17
27.63
71.53
60.55
74.55
31.27
(e)
71.09
43.72
132.50
28.06
74.74
60.32
74.74
31.29
Conclusion
In this work, present Stitch and Tell, simple and scalable data augmentation strategy that injects spatial structure into vision-language training. SiTe combines image stitching with spatially- aware caption generation to provide weak spatial supervision without requiring human annotation architectural changes. We apply SiTe to multiple backbones, including LLaVA, Qwen2 and HALVA. across two training datasets and thirteen benchmarks. Experiments show consistent improvements on both spatial understanding and general vision-language tasks. These results demonstrate that encoding spatial structure into training data can improve tross-modal alignment and spatial reasoning, while maintaining strong general performance. We hope this work provides a lightweight and broadly applicable approach structured multimodal data augmentation for spatial understanding.
10

---

References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: Visual Question Answering. International Conference on Computer Vision (ICCV), 2015.
[2]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabís, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofrr Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
[3] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14455-144 14465, 2024.
[4] Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Advances in Neural 2023.
Information Processing Systems,
[5] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[6] Phillip Chlap, Hang Min, Nym Vandenberg, Jason Dowling, Lois Holloway, and Annette Ha- worth. A review of medical image data augmentation techniques for deep learning applications. Journal of medical imaging and radiation oncology, 65(5))544-556)) 2021.
[7] Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfci Yang, Afshin Dehghan, et al. Mm-spatial: Exploring 3d spatial understanding in multimodal 1lms. arXiv preprint arXiv:2503.13111 2025.
[8] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 1469-88888 PMLR, 2023.
[9] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion- based augmentation. Advances in neural information processing systems, 36:79024-79034, 2023.
[10] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for visual understanding and reasoning. arXiv preprint arXiv:2403. 1401, 2024.
[11] Tejas Gokhale. Towards robust visual understanding: A paradigm shift in computer vision from recognition to reasoning, Al Mag., 45(3):429-433 2024.

---

[12] Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mixgen: new modal-modal data augmentation. In IEEE/CVF Winter Conference on Applica- lions of Computer Vision Workshops, WACV 2023 Workshops, Waikoloa, HI, USA, January 3-7, 2023, pages 379-389. IEEE, 2023.
[13] Guillermo Iglesias, Edgar Talavera, Ângel González-Prietes Alberto Mozo, and Sandra Gómez- Canaval. Data augmentation techniques in time series domain: a survey and taxonomy. Neural Computing and Applications, (01133)001231000 2023..
[14] Lisa Jöckel, Michael Kläs, and Silverio Martinez-Feemánderr Safe traffic sign recognition through data augmentation for autonomous vehicles software. In 2019 IEEE 19th International Conference Software Quality, Reliability and Security Companion (QRS-C), pages 540-541. IEEE, 2019.
[15] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[16] Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yunchao Wei. Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. CoRR, hbbs/2308.10253. 2023.
[17] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Coco2017 dataset, jan 2025.
[18] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755, Springer, 2014.
[19] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Trans. Assoc. Comput. Linguistics, 1 11:635-651, 2023.
[20] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 06296-26306, 2024.
[21] Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint 2xXi::2407.06866 2024.
[22] Abby O' N'illl et al. Open x-embodiment: Robotic learning datasets and RT-X models : Open x-embodiment collaboration. In IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13,17, 2024, pages 6892-6903, IEEE, 2024.
[23] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011
[24] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1
Shawe-Taylor, million captioned photographs. J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24, Curran Associates, Inc., 2011.
[25) Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S Ryoo, and Tsung- Yu Lin. Learning to localize objects improves spatial reasoning in visual-Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12977-12987 2024.
[26] RANJAN Sapkota, SHAINA Raza, MAGED Shoman, A Paudel, and M Karkee. Multimodal large language models for image, text, and speech data augmentation: A survey. arXiv preprint 2511 2025.
12

---

[27] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ank. and Tomas Pfister. Data- mata-augmented phrase-level alignment for mitigating object hallucination. arXiv preprint arXiv:2405. 18654, 2024.
[28] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
[29] Manli Shu, Yu Shen, Ming C Lin, and Tom Goldstein. Adversarial differentiable data aug- mentation for autonomous systems. In 2021 IEEE international conference on robotics and automation (ICRA), pages 14069-14075. IEEE, 2021.
[30] Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d rision-language models for robotics. CoRR, abs/241 1. 16537, 2024.
[31] Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard, editors, Conference Robot Learning, 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Muchine Learning Reseurch, pages 1998-4726.. PMLR, 2024.
[32]
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Iyer, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[33] Akshar Tumu and Parisa Kordjamshidi. Exploring spatial language grounding through referring expressions. arXiv preprint arXiv:2502.04359, 2025.
[34] Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang. Sharon Li, and Neel Joshi. Is A picture worth A thousand words? delving into spatial reasoning for vision language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Anmual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[35] Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. A survey on data synthesis and augmentation for large language models. CoRR, abs/2410. 12896, 2024.
[36] Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25- March 4, 2025, Philudelphia, PA, USA, pages 7907-7915. AAAI Press, 2025.
[37] Penghao Wu and Saining Xie. V*: Guided visual search llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084 13094, 2024.
mechanism in multimodal
[38] X.AI. Realworidqa. Blog post, July 2025. Accessed: 2025-07-30.
[39] Yutaro Yamada. Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and llker Yildirim. Evaluat- ing spatial understanding of large language models. Trans. Mach. Learn. Res., 2024, 2024.
[40] Jihan Yang. Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. CORR, 14171, 2024.
13

---

[41] Toby Yang. LLaVA-Qwenz. Integrating qwen2 with 1lava for multimodal pretraining and chat. https:// gi thub. comm/Tooyaagg//Laaaaaeeeee
2024. Accessed: 2025-05-12.
[42] Hang Yin, Zhifeng Lin, Xin Liu, Bin Sun, and Kan Li. Do multimodal language models really understand direction? benchmark for compass direction reasoning. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
1-5, 2025.
[43] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67-78, 2014.
[44] Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where look: Training-free free perception of small visual details with multimodal llms. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,
2025. OpenReview.net, 2025.
[45] Haoyi Zhu, Honghui Yang, Yating Wang, Jiange Yang, Limin Wang, and Tong He. SPA: 3d spatial-awareness enables effective embodied representation. CoRR, abs/2410.08208, 2024.
[46] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine. Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marein Choromanski. Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: lasion-language-actie
models transfer web knowledge to robotic control. In Jie Tan, Mare Toussaint, and Kourosh Darvish, editors, Conference on Robot Learning. CoRL. 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pages 1165-2183. PMLR, 2023.
14

---

A Limitation
While Stitch and Tell demonstrates strong performance and scalability across multiple settings, it currently focuses on simple binary spatial confgurations--prinaat horizontal (left-right) and vertical (top-down) compositions. This design enables efficient supervision generation and robust model alignment, but may limit the model's exposure to more complex spatial relationships that occur real-world environments, such as diagonal layouts, circular arrangements, or relative distances in 3D space. We believe extending SiTe to handle richer spatial topologies, possibly by incorporating depth maps, multi-image compositions, or 3D-aware stitching strategies, promising direction for future work.
B Data Template
To support structured data generation in our Stitch and Tell method, we employ a collection of spatially guided templates that are automatically generated and manually filtered for diversity and clarity by GPT-4o. These templates are designed to inject explicit spatial structure into image-text data and are categorized follows:
Spatial Caption Templates: As shown in Table 6 and Table 7 collect 35 templates for horizontal stitching and 29 for vertical stitching. These templates are automatically produced using a language model and then curated ensure naturalness and correctness. Each template organizes two independent captions into single sentence with clear spatial cues, such as "Lefi side displays: [captionl), J. Right side shows: (caption2)) The diversity of expressions encourages the model to learn generalized spatial grounding rather than relying on specific lexical patterns.
As shown Table and Table 9 we generate 20 question templates in each stitching way (horizontal or vertical). These templates used to form weakly supervised question answer pairs such as "From the observer's viewpoint, is the cat on the left of the car?", with answers derived directly from the known image layout. Templates were automatically generated with instruction-uun models and then filtered to preserve clarity, directional accuracy, and grammatical diversity.
Spatial QA Templates:
15

---

Table Caption templates used for constructing structured spatial descriptions in S&T. Placeholders [captioni] and [caption2) represent the original descriptions from the left and right images, respectively.
Template
1. On the left, {caption1}. Meanwhile, the right side presents {caption2).
This composite image showcases {caption1) on the left, contrasting beautifully with {caption2) the right.
The left section highlights {caption1), while the right side draws attention to (caption2).
4. Displayed in the left half's is {caption1), while the right half features {caption2)..
[captioni]
left and {caption2] on the right.
Left image shows: (caption1] Right image shows: (caption2).
7.
On the left: {caption1} II On the right: {caption2).
8.
First image description: {caption1), Second image description: {caption2).
9.
Image pair - Left: {caption1> Right: [caption2).
10.
[Left] (caption1) [Right] (caption2).
11.
Left side displays: {caption1) <=> Right side displays: (caption2).
12.
Left view: {caption1] Right view: [caption2).
13.
Left portion: {caption1> Right portion: {caption2).
14.
Left panel shows: {captioni} II Right panel shows: [caption2).
15.
Left section: [caption1] <-> Right section: {caption2).
16.
The right one shows [caption2), while the left displays (caption1}.
17.
pair images: on right we see (caption2), on the left there's {caption1).
18.
The right image contains {caption2), paired with left image showing {caption1}.
19.
Two scenes: [caption2] the right, accompanied by {captioni> on the left.
20.
Right image depicts {caption2), contrasting with the left image showing {caption1).
21.
[caption1] (left) vs (caption2) (right).
22.
Contrast perspective presents [caption1), while right shows {caption2).
Visual contrast: shows {caption1), Right shows {caption2).
24.
Left portrays [caption1), Right highlights {caption2).
25.
Left vs Right: (caption1} & {caption2).
26.
Left illustrates (caption1), whereas right (caption2).
27.
Left image showcasing [caption1], and right image featuring [caption2).
[caption1] and {caption2].
(caption1), {caption2).
30.
Left: (caption1) Right: (caption2).
31.
[caption1] {caption2).
32.
Left/Right: (caption1)/ (caption2).
33.
[caption1] vs {caption2))
16

---

Table 7: Templates used for generating structured captions in top-down stitched images. Placeholders {captioni] and {caption2) represent captions from the top and bottom image regions.
No.
Caption Template
1.
On the top, you (caption1), the bottom side presents {caption2).
2.
This composite image showeases {caption1) the top, with (caption2) the bottom.
3.
The top section highlights (caption1), while the bottom side draws attention to (caption2), creating engaging visual comparison.
4. Displayed in the upper half is (caption1), while the lower half features (caption2), illustrating unique attributes side by side.
5. A striking juxtaposition: {captioni) on the top and (caption2) on the bottom, offering an interest- ing visual narrative.
6. Top image shows: {caption]), Bottom image shows: {caption2).
7.
On the top: (caption1). On the bottom: {caption2).
First image description: [caption1), Second image description: {caption2).
Top: {caption1); Bottom: {caption2).
10.
Top part display: (caption1) Bottom part display: (caption2).
11.
Upper displays: {caption1), Lower displays: (caption2).
12.
Top view: {captioni), and Bottom view: {caption2).
13.
Top portion: {caption1} > Bottom portion: [caption2).
14.
Upper panel shows: (captioni} Lower panel shows: (caption2).
15.
Upper section: {caption1} <-> Lower section: {caption2).
16.
The bottom shows {caption2), while the top displays (caption1).
17.
A pair of images: the bottom see (caption2), and the top there's (caption1).
18.
The bottom image contains {caption2), the top image showing {caption1}.
19.
Two scenes: [caption2) on the lower side, accompanied by {caption1] on the upper.
20.
Bottom image depicts (caption2), contrasting with the top image which shows (caption1).
21.
[caption1] (top) vs (caption2) (bottom).
22.
Top side presents [caption1], while hottom side shows (caption2).
23.
Top shows {caption1), Bottom shows (caption2).
24.
Top portrays {captioni), Bottom highlights (caption2).
25.
Top vs Bottom: (caption1} & (caption2).
26.
Top illustrates {caption1), whereas bottom captures (caption2) detail.
27.
A split view: top image showcasing {caption1), and bottom image featuring (caption2).
28.
[caption1} and {caption2].
29.
[caption1), {caption2).
30.
Top: (caption1), Bottom: (caption2).
17

---

Table 8: List of spatial QA templates used in instruction tuning. Each template is instantiated with [left_obj} and {ri (right_obji and paired with binary answer.
Template (with placeholders)
Answer
1. From the observer's point of view, is [left_obj} located to the left of Yes {right_ob]]
2. From the camera's viewpoint, [left_obj]?
(right_obj) the right of Yes 3. Looking from the front, is (left_obj) placed to the right of (right_ob])
No
4. As observed from the viewer's perspective, does (right_obj} appear left of No (left_obj]?
5. From the point of view of the observer, is [left_obj} on the left side of (right_obj]?
Yes
6.
{right_obj); from the camera's perspective, situated to the right of [left_obj}?
Yes
7. When facing the image, does the left contain (left_obj} and the right contain {right_objis
Yes
8. From frontal viewpoint, is (left_obj) the left side of (right_obj)
Yes
9. As seen in the combined image, is (right_ obj} located right of (left_obj)
Yes
10. To the viewer, does [left_obj] appear on the right side of (right_obj}?
No
11.
Does (left_obj} appear on the left side of (right_obj)?
Yes
{right_obj} located to left of {left_obj}?
No
13.
Can (left_obj] be found the right of (right_obj}??
No
14.
Is fright_obj) positioned on the right of (left_obj}?
Yes
15.
Would you say {left_obj} is left to (right_obj}?
Yes
16.
{right_obj} on the left [left_obj} in this composition?
No
17.
Does the image on the left contain (left_obj} while the right image contains (right_obj}?
Yes
18.
(left_obj} positioned on the right (right_obj} instead?
No
19.
this pair, (right_obj) located to right (left_obj]?
Yes
20.
you observe carefully, is fleft_ob) the right and (right_obj} the left?
No
18

---

Table 9: Templates for generating top down spatial question question-answer answer (QA) pairs. Placeholders [top_obj] and {bottom_obj} denote entities from the top and bottom image regions.
No. QA Template
Answer
1. In the image, is {top_obj} located above (bottom_obj)?
Yes
From the viewpoint of the ohserver, is {bottom_ poootom_obj}> below {top_obj}?
Yes
Would you say that {top_obj} is placed underneath (bottom_obj}?
As observed from the front, [bottom_objj situated on top of (top_obj}?
5. Does the top part of the image contain {top_obj} while the bottom part has {bottom_obj)?
Yes
6. Looking from to bottom, do you first (top_obj), then (bottom_obj)?
Yes
7. Is (bottom_ob) objtt placed above (top_obj} in this composition?
No
From top-down perspective, is [top_obj}> above {bottom_obj}?
Yes
this combined image, does {top_obj} appear at the top and {bottom_obj} the bottom?
Yes
10. (top_obj} below {bottom_objj}?
11. Would you say (bottom_obj)
lower vertical position than {top_obj}?
Yes
12. Does the vertical layout place [top_obj} higher than {bottom_obj}?
Yes
13. Is (bottom_ob) _bbitom__bbi>>>o appearing above [top_obj} in this image?
No
14. Do you see (top_obj} on the upper half and {bottom_obj] on the lower half of the image?
Yes
15. Can we find {bottom_obj} positioned higher than {top_obj} the image?
16. In vertical layout, is (top_obj} stacked above (bottom_obj}?
Yes
17. Does (top_obj) sit at the bottom while (bottom_obj) is top?
No
18. Is the object {top_obj} visually located above {bottom_obj} from this angle?
Yes
19. Would you agree that (bottom_obj] is hencath [top_obj} in this view?
Yes
20. Is (bottom_obj} placed at the upper portion the image, above {top_obj}?
No
19

---

C Additional Experiments and Results
C.1 Ablation study
Table 11: Performance of MaaaV--Qee2--- 1.5B pretrained the 558K dataset under different spatial data augmentation strategies. Results are reported on spatial understanding benchmarks (left) and general vision-language benchmarks (right).
Setting
Spatial Understanding
General AsionnLangguage
COCO-QAspunic Spotial
Spatial-MM MMEpostionn
Vetspu
COCO-QA
VQA,2
MMB
MM-V Vet
62.57
41.25
63.00
22.90
65.26
54.18
66.68
22.10
(c)
60.88
41.08
63.20
20.80
64.81
53.50
67.51
21.04
(d)
59.83
41.11
64.00
21.75
62.74
52.83
70.32
21.30
53.20
39.66
55.00
18.15
54.74
46.80
64.47
17.10
Table 12: Performance of LLaVA-v1.5-7B pretrained on the Flickr30K dataset under different spatial data augmentation strategies. Results are reported on spatial understanding benchmarks (left) and general vision lision-language benchmarks (right).
Setting
Spatial Understanding
General Vision-Language
COCO-QA
Spatial-MN MMEresioon
MM- Vetspu
COCO-QA
VQAv2
MMB
MM- Vet
71.42
44.97
130.70
26.20
73.74
59.73
72.88
29.59
(1)
69,80
42.96
130.30
27.54
72.53
59.65
72.56
31.06
(2)
69.63
44.50
126.90
27.59
72.44
59.10
72.42
29.40
71.03
44.21
133.00
26.17
73.28
58.73
72.31
29.19
Table 13: Performance of LLavA-Qwen22-1.. pretrained on the Flickr30K dataset under different spatial data augmentation strategies. Results are reported on spatial understanding benchmarks (left) and general vision- vision-language language benchmarks (right).
Setting
Spatial Understanding
General Vision-Lane Vision-Language
COCO-QA SScc--QAspniall
Spanial
Spatial-MM MMEpusisonn
MM- Vetspu
COCO-QA
VQA,2
MMB
MM-Ves Vet
47.71
39.35
61.00
12.47
48 18.68
42.95
51.33
10.80
(1)
47.57
39.28
57.60
12.00
48.17
40.78
49.43
10.54
(2)
48.16
38.88
56.50
12.60
48.76
41.69
52.04
11.23
43.06
39.63
60.00
11.65
45.72
39.36
49.09
10.53
To identify the optimal stitching ratio for spatial data augmentation, perform ablation experi- ments on four settings: LLaVA-v1.55 -- trained on 558K and Flickr30K, Qwen2-1.5B trained on 558K and Flickr30K. In each setting, we vary the number of stitched samples and report perfor- mance on both spatial understanding and general vision-language benchmarks.
Table 10: Settings of SiTe variants used for ab- lation study during pretraining on the Flickr30K dataset. Each variant uses different ratio X, which denotes the proportion of stitched samples to the remaining samples in the training set. The total number training data varies accord- ingly.
Setting
Images (Total data size)
A
In the main text, we present ablation results our SiTe method on LLaVA using the 558K dataset. Here, we provide additional ablations on two new settings: ALaVA-v1..5-77 pretrained on Flickr30K. and LLaVA-Qwen2-IL.55 pretrained on both 558K and Flickr30K.
SiTeactutt
24K
:: 3.0
(1) N=IK
28K
1: 13.0
(2) N-SK
20K
1: 1.0
N=7K
16K
1 0.14
For the 558K dataset, we evaluate Qwen2 under four configurations with different stitching sizes: N = 10K (setting (c)), N = 50K (default setting on 558K), N 100K (setting (d)), and N 139K (setting (e)). The corresponding results are summarized in Table [I
20

---

For Flickr30K, we evaluate on N = 1K (setting (1)), N = 3K (default setting on Fl Flickr30K), N 5K (setting (2)), and N 7K (setting (3)). Dataset statistics are shown Table I0 and the results are reported in Table 12 and Table 13 for LLaVA and Qwen2 backbones.
We summarize key observations as follows:
Qwen2 on 558K (Table The default 1:3 setting achieves the best overall spatial performance. Specifically, it yields the highest results on COCO-QAspu (62.57), Spatial-MM (41.25), MM- Vetspar (22.90), and also achieve most tops general metrics. Increasing the stitching ratio beyond this point (e.g., settings (d) and (e)) leads to degradation on both spatial and general benchmarks, indicating over-augmentation.
LLaVA on Flickr30K (Table[T):: The 1:3 setting (SiTeaefauit) still consistently provides strong performance across tasks. It outperforms other ratios on Spatial-MM (44.97), MMEposiion (130. 70), and general rision-languse tasks like COCO-QA (73.74) and MMBench (72.88). Other ratios (settings (3) offer marginal or inconsistent gains, and in some cases hurt generalization.
Qwen2 on Flickr30K (Table 13): In this setting, the 1:3 setting still maintains strong spatial performance, achieving the best results on MMEPosition (61.00 (61.00) and balanced results on general tasks. While setting (2) gives slightly better COCO-QA and MMB scores, the default ratio still offers stable and robust trade-off.
In summary, the 1:3 stitch-to-raw ratio consistently provides balanced trade-off between spatial reasoning and general vision-language performance. Ratios that are too low underutilize the spatial supervision potential of SiTe while overly high ratios risk oversaturating the model with structured spatial prompts, which may harm generalization. Based on these findings, we adopt the 1:3 setting as the default configuration throughout our experiments.
In all experiments, we define the default SiTe configuration as the one where the stitching-to -o-bang sample ratio is approximately 1:3, which provides a good trade-off performance between spatial understanding and general vision language tasks.
C.2 More Qualitative Analysis
In this section, we present additional qualitative examples to illustrate both the effectiveness and limitations of the proposed method. LLaVA refers the baseline model. piTeeretrain indicates the model trained with stitched image-caption pairs during the pretraining stage, while SiTesrr refers to the model fine-tuned using structured spatial question-answer pairs.
As shown in Case 1 and Case 2, although all models attend to the relevant objects (c.g., the dog and the people), the baseline model fails to correctly answer the spatial question due to limited understanding of directional language, selecting an incorrect option such as "A. bottom". In contrast, both SfTe-enhance models correctly interpret the spatial relation and make the right prediction. Case 2 also involves a challenging cumera-perspecttee reason from the observer's viewpoint. Here, both SiTe variants successfully leverage spatial knowledge injected through pretraining and fine-tuning, resulting in correct answers.
transformation, which often requires the model
scenarios. In Case 3 and Case 4, while all models attend to the appropriate regions (c.g., the glasses in Case 3 and the phone in Case 4), they fail to answer correctly. This suggests that despite improved attention, the models still struggle to generalize spatial understanding across different viewpoints.
However, limitations remain in non-camera-perspece
These examples highlight both the strengths and boundaries of the proposed method: SiTe im- proves spatial reasoning under observer-aligne prompts, but further work needed enhance generalization under varied spatial perspectives.
C.3 Standard Deviation of Main Results
We compute the standard deviation main results in Table As indicated by the standard deviation results, our performance gains are consistent and robust.
21

---

Where dogs from the people's perspective? bottom right front D. left
Case
Sileerri
Which hand of the boy is the holding? A. left B. right
Case
LLaVA
SiTepwerr A
Silegrs
Which hand is the man using to hold the glasses? A. left right
Case 3
LLaVA:
SiTewarai
Silesrs
which hand is the man holding the phone? A. left right
Cuse
LLaVA:A
SiTawasx:
SiTesm
Figure 5: Qualitative comparisons among the baseline LLaVA and our SiTe models.
22

---

Table 14: Standard deviation of main results
Model
Spatial Understanding
General Vision Language
COCO QAapei (%) Sputial S)
MMEess
Vetspai
COCO (QA,z(%) MMB(S)
MM-Vet
Protesising Stoge
VAa
67.7240..00 10
42.02:20.54
127
26.52:000
70.52:0.332
60.4220116
16
73.50 31.11 11:11111
LLaVAgaue
68.09-00177
43.01-:0.11
29440-00088
71.36-+0.11
60.604-.17
74006-00055
20-012-
LLAVAcmp
67,8240225
42.69:20.24
1277742004
28.53900112
7098
60.57:00224
74.41
LLLVAysR
67,85-40221
0,365
121.6741011
29442400.54
705004-0.28
60.57440225
73.88-00.44 31.47
LLaVAsTeraad
68.75-20.06
43.78:20.14
133332043
28.43:0.00 19
71.54:0.27
74.45::0:5 21.00:00344
LLaVAgTe-rati
20.06-0.318
44.15:10.22
15280-0077
28.08--0.166
71.19440.31
60.57:00.00
73.64-20.15 32.27-00.10
LLaVA dicke
68.664221
12900:025
71.75::0228
59.634..19
72.33:0.13
LLLVA
68.03---.15
4481:10.19
129.4440.78
25.86-1-0.11
71,1240::2
59.42400.18
0.136-10.14 29.50
LLaVA
69.34:1:.44
42.69:20211
131.11:1::2
28.73:1:0.10 10
72.12:10.26
60.20100211
72.68:20117 29.87::.001
LLAVATSE
68.26200188
42.70:20.28
1287550007
26.18:b0.13
71:3240.31
59.9420.244
72.87:20.19 30.45:00.18
LLaYA
11,42-0.13
73.34:0229
LLaVA
SaTe-ratie
71.5150155
44.31:0.255
151.5000111
28.00:0.12
73.89:00225
59.3420..00
01.66:00:16 30.97
quacinr
62.25-00.16
40.5500.18
60754-0.3
20.72400.10
64.72440:22
53.66-00.25
61.67:00.11
22.48-1-0.1
Qwentqueate
60.73400114
40.78:20221
60504067
1962
60222:0.26
52.77::0.22
66.61 0:2225 20:97::00009
Qwen22iog
62,06-1021
40.68:0.155
61254-058
20.55 +-111
64.7440.19
53.28--0.27
67.45 0.21 22.52400.14
Qwen2v3R
57.18550.18
01.11
583040773
22.40:0.16
16
60222:0.28
54.4420.23
66.58:20.22 22.95-20.12
Qwen2
62.57-1-.15
41.25:00:22
63000061
2250--0.14
65:26:4-023
54.18400220
66.68--00. 0.19 02.13
Quen2
62.52-60.17
41.00:20.19
68:04:388
21.00:0.13
65.10:40.7
54.25-00.21
67.57520.20 22.80022222200002 22.80-20.12
Qwenz
47.1020.30
08.0
51.25:0644
10.520.00 10
47.00:20.31
42.38::0.25
0.190 10.50:20.09
Owenz
40.08-5:023
29.10:20:77
5833550075
9.504-.13
42.45::0.29
39.04:::225
07.86:20.21
9.20--0.10
Qwen Slickr
42.37-20.22
39.3720.24
642555078
830.0.122
44.61:10.28
40.11202255
46.43.20222
10:0020.11
46.00:1..19
30.9020.200
67
10.15:0.11
48.00:1::66
40.9110:24
20.71
0.60
Qwen
47.71:10.17
19.35:20,18
61.67:1882
12.4710.14
48.68:10.25
42.95:0221
51.38:10.18 10.8010.11
Qwenz
Bicke
49.04-0.15
40.04:00200
56:00::000
13.10+0115
47.38:0027
43.9120.233
51.31 10.6020.10
Supervised Fine-toning Stage
LLaVA
T-rannd
68.8120366
43.16:20.27
128.001008
27.06:0.11
71.7420477
61.172:0.9
04.6 2626 31
LLaVA
68.154-0211
46.6:20023
1360040522
29.0040.14
71.3240.33
60.24:222
73.374-0.19 31.50-1-.10
28.10:1::23
LLaVA
Eteratie
68.3740022
48 08.58:00.66
141.00 1.03 31.05:0000 16
70.02-0.888
60.87:0:225
73.76::0.20
12.15
Qwen2
Teeand
58.53-10047
41.24:20033
65:3311:554
23.10:1::44
61.77:1..22
54.58:10003
66.37:20.11 23.93:1::22
QwenZ
$9.66.0.33
42.31:0:27
6275::063
24.07:0.11
62.59.1.335
55.3020.25
65.75-00119 22.8110.11
Qwen Perrand
59.95-40.53
42.22:00.11
61.67400.3
23:20--0.3)
63.0041.110
54.75-00.14
66.43-6-013 23.93-40.19
Qwent Quenisye-ratt
60.22:1:0.35
41.56:10.29
$4.50110.59
25.1011::200
62.86:10411
$4.77:10.20
67.00:20.18 24.27:10.15
HALYAguedie
63.16-1:0.14
43.07:10.26
HALVASTT
64.77-500.18
44.15:20.22
123333+033
26.10:0.11
68.54400.77
61.03:2:.200
71.54::0.15
30.80:00110
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the core motivation and contribution of the work-namely that current vision-language models struggle with spatial understand- ing due to limited spatial supervision training data. The proposed method, Stitch and Tell (SiTe), is presented as a simple and efficient data augmentation strategy that injects spatial structure without requiring labels or additional model generation. The claims are well supported by experiments across multiple benchmarks and architectures. Both the strengths (lightweight, scalable, effective across stages) are presented a balanced manner, matching the scope and results of the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the SiTe limitation Appendix Limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section Experiment Setup, we introduced the parameters and environment required for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
25

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification:
We will release the data and code soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips cc/ wtc/publicc// cobollccggiddsss gwt.///ieee//eee// Codeeoomiiiieeee descoeuuuud SanSSuubissss iolbbmisssiooi ssisnnoiccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section Experiment Setup, we introduced the parameters and environment required for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the standard deviation main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section Experiment Setup, we introduced the parameters and environment required for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics bestpa://neuripeee http:/rrpp.....bbbblc//cccceeeee
Answer: [Yes]
Justification:
We sure preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: CC-BY 4.0
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "p3Sxx6QdO9", "reviewer": "Reviewer_3L35", "summary": "The paper introduces Stitch and Tell (SiTe), a novel, annotation-free, and lightweight multimodal data augmentation method designed to enhance spatial understanding in vision-language models. The core issue addressed is spatial hallucination, in which the models generate incorrect spatial descriptions of objects due to sparse spatial information in training text. \nThe approach aims to generate a new training set with spatial information, defining two steps:  \n\n1. Image-Spatial Stitching: Two images are stitched either horizontally or vertically, creating an image with an explicit spatial layout.\n2. Text-Spatial Telling: New spatially-aware captions and question-answer (QA) pairs are generated based on the relative positions of objects in the stitched image.\n\n\nInterestingly, the approach avoids the need for expensive human annotations or model-generated data, making it scalable and efficient.\n\nSiTe is evaluated across multiple model architectures and two training datasets, in which multiple conditions are generated to show the effectiveness of the approach. Results show that SiTe significantly improves in the spatial evaluation in comparison to the baseline and  maintains general vision-language effectiveness", "questions": "See the weaknesses.", "limitations": "Authors do not include any section about limitations. \nYou can find the limitations of the paper in weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- **Clear Problem Motivation**. The paper addresses a well-defined and significant problem: the insufficient representation of spatial information in existing vision-language datasets. By quantitatively analyzing this limitation, the authors effectively motivate the need for improved spatial understanding in vision-language models.\n\n- **Novel, Simple, and Scalable Method.**  The proposed approach, SiTe, introduces a novel yet simple data augmentation technique aimed at enhancing spatial reasoning. Notably, the method requires no modifications to existing model architectures and incurs minimal computational overhead, making it highly scalable and easy to integrate into current workflows.\n\n- **Comprehensive Evaluation.** The effectiveness of SiTe is validated through extensive experiments across a range of model architectures and benchmark datasets, demonstrating the effectiveness of the proposed method against the baseline..\n\n**Weaknesses**\n\n**Weaknesses of the Methodology and Approach.** \n\n- **Unclear Image Pair Selection Process**. The process for selecting image pairs from the dataset is insufficiently described, making it difficult to reproduce the experimental setup. For the sake of replicability and clarity, it would be beneficial for the authors to explicitly detail the selection criteria and sampling procedure. Additionally, it remains unclear whether the semantic similarity or object co-occurrence within image pairs has any influence on the model’s learning dynamics. This aspect warrants further discussion, as it could significantly affect the training signal and generalization capabilities.\n\n- **Limited Scope of Spatial Relations.** The proposed method focuses exclusively on binary spatial relations, specifically left/right and above/below. While this targeted approach is not inherently a weakness, the paper does not acknowledge this limitation or provide a discussion on the method's potential to generalize to a broader set of spatial relations (e.g., inside, behind, near). Addressing this gap would strengthen the paper's impact and suggest avenues for future work.\n\n**Weaknesses in Evaluation and Experimental Design**\n\n- **Insufficient Detail on Evaluation Settings.**  The evaluation protocol lacks critical detail, particularly concerning the alignment between the nature of the training and evaluation data. While the proposed method is designed to enhance the understanding of explicit spatial relations, the pretraining dataset (VSR) predominantly contains implicit ones. Moreover, the paper does not clarify whether the evaluation datasets are specifically constructed or filtered to target explicit spatial reasoning. This ambiguity makes it difficult to assess the actual efficacy and relevance of the SiTe augmentation strategy.\n\n- **Table 2 Lacks Clarity**.  Table 2 is difficult to interpret due to insufficient description of the training setups for each model variant. For example, the distinction between LLAVA_SiTe and LLAVA_{SiTe}^{Flickr} is unclear: does the former involve pretraining on both the 558K and Flickr30k datasets, or only on the former? Furthermore, the MMEpos scores reported in the table exceed 100, which raises concerns about potential reporting or normalization errors. These issues significantly hinder the reader’s ability to interpret and compare results accurately.\n\n- **Unexpected Competitiveness of Baselines**. Surprisingly, classical data augmentation methods, despite not being tailored for spatial understanding, perform comparably to the proposed method. This finding potentially undermines the strength of the SiTe approach and its specific contribution to spatial reasoning. To more rigorously validate the benefits of the proposed method, it would be appropriate to include comparisons with state-of-the-art techniques explicitly designed for spatial relation learning.\n\n- **Marginal Gains from Supervised Fine-Tuning**. The supervised fine-tuning stage does not appear to yield substantial improvements over the pretraining phase. This calls into question the effectiveness and necessity of this additional step within the overall training pipeline. A deeper analysis or ablation would be helpful to understand the role and impact of fine-tuning in the context of the proposed method.", "was_revised": true, "final_justification": "I read the rebuttal and the rest of the review, and I decided to change my scores.", "rebuttal": "**Dear Reviewer 3L35, thank you for recognizing the strengths of our paper. We are pleased that you approve the motivation of our work, appreciate the novelty and extensibility of the SiTe method, and acknowledge the comprehensive experiments. We will address your concerns and questions below.**\n\n---\n> **`Q1: Can you describe the process for selecting image pairs, and whether object co-occurrence within image pairs influences the model’s learning dynamics?`**\n\nR1: Thank you for your question. The process for selecting image pairs is thoroughly described in Section 3 (“Image-Spatial Stitching”) and Section 4.2 (“Pretraining Stage”) of our paper. During the image-spatial stitching, we perform random sampling as follows:\n\n*Given two randomly sampled image–caption pairs (I₁, T₁) and (I₂, T₂) from the dataset, we first construct a stitched image by spatially stitching the two input images along a specific axis*\n\nAlgorithm 1 provides detailed pseudocode for the image stitching procedure, including how the canvas is created and how the two images are merged.\n\nTo avoid duplicating image-text information, we describe in Section 4.2 a non-overlapping sampling strategy. \n\nWe use the raw dataset to create both horizontally and vertically stitched samples. In the final training set, there are three components: horizontally stitched samples, vertically stitched samples, and the remaining raw data. Each image–caption pair used for stitching is removed from the raw dataset and does not appear as a standalone example.\n\n*To avoid duplicate supervision, image–caption pairs used for stitching are removed from the raw set, ensuring each image appears only once.*\n\nThis ensures that the model sees exactly the same underlying image-text information as the baseline, making comparisons fair.\nWe also maintain a 1:1 ratio of horizontal to vertical stitching in the training set:\n\n*We apply both horizontal and vertical stitching in a 1:1 ratio.*\n\nAnd we specify the default setting:\n\n*By default, the stitched data make up one-third of the total set.*\n\n**These steps support the full reproducibility of our experiments.**\n\nAdditionally, we implement an object co-occurrence filter. As described in Section 3 (“Tell the Question and Answer”), we remove overlapping objects when constructing spatially-aware QA pairs:\n\n*To reduce ambiguity, we remove overlapping nouns shared by both sides.*\n\n**This design ensures that SiTe produces unambiguous spatial questions.**\n\n---\n> **`Q2：The authors do not discuss limitations and the potential for the method to generalize to a broader range of spatial relations`**\n\nR2: We would like to clarify that the limitations are discussed in Appendix A. Our work currently focuses on horizontal and vertical spatial configurations and we will exploring broader spatial relations, including 3D perception in future work.\n\nTo further address your concern, we evaluated our models on the CV-Bench[a], which specifically test for complex spatial reasoning including a broader set of spatial relations. \n\nThe results are shown in the table below (Owing to space constraints, further results on additional datasets are provided in our response to R4 for Reviewer eNJr.):\n|Model|CV-Bench 2d|CV-Bench 3d|\n|--|--|--|\n|$\\mathbf{LLaVA_{Baseline}}$|52.56%|33.43%|\n|$\\mathbf{LLaVA_{SiTe}}$|**54.28%**| **35.50%**|\n|$\\mathbf{Qwen_{Baseline}}$|46.77%|47.63%|\n|$\\mathbf{Qwen_{SiTe}}$|**48.16%**|**50.36%**|\n\n**As shown, the injection of spatial supervision via SiTe leads to demonstrate strong generalization to more complex spatial reasoning scenarios.** A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships.\n\n[a] Tong, Peter, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" NeurIPS 2024.\n\n---\n> **`Q3: It is unclear whether the evaluation datasets are specifically constructed or filtered for explicit spatial reasoning. While the VSR dataset contains implicit spatial relations, is it fair to compare?`**\n\nR3: Thank you for your comment. We provide a clear description of the evaluation benchmarks in Section 4.4, where we introduce each dataset and its characteristics. The pretraining datasets including those for stitching, are described in detail as follows:\n\n*To evaluate the effectiveness of SiTe across different training sets, we experiment with two image–caption datasets: blip_laion_cc_sbu_558K(558K) and Flickr30K.*\n\nSiTe uses horizontal and vertical spatial relations, with templates listed in Appendix Tables 5–8. Nevertheless, the evaluation benchmark are public that cover a much broader range of spatial relations and were **not filtered for** explicit spatial reasoning. \n\nIt is fair to compare the $\\mathbf{LLAVA\\_{SiTe}}$ and  $\\mathbf{LLAVA\\_{VSR}}$. The VSR dataset covers many spatial relation types. We use it as a strong baseline in our comparisons. As shown in the table of the top 10 relations below, VSR has not only basic directional relations but also includes other types. Our results show that SiTe outperforms models trained with VSR data ($\\mathbf{LLaVA_{VSR}}$), highlighting the effectiveness of our approach.\n\n|**TOP 10 Relations**|**Prob**|\n|--|--|\n|touching|11.89%|\n|in front of|7.01%|\n|behind|6.31%|\n|on|5.49%|\n|under|4.93%|\n|at the right side of|4.28%|\n|on top of|4.05%|\n|at the left side of|3.86%|\n|beneath|3.24%|\n|contains|3.01%|\n\n---\n> **`Q4: What is the difference between LLAVA\\_SiTe and LLAVA\\_{SiTe}^{Flickr}? And why does the MME score exceed 100?`**\n\nR4: Sorry for the confusion! We respectfully clarify that, $\\mathbf{LLaVA_{SiTe}}$ is pretrained on the original LLaVA training set (558K) with 1/3 data by SiTe, while $\\mathbf{LLAVA\\_{SiTe}^{Flickr}}$ is pretrained only on Flickr30K with 1/3 data by SiTe. The motivation for including both settings is to demonstrate the generalizability of SiTe in different datasets.\n\nRegarding the MME benchmark [a], each image is associated with two questions: one for \"accuracy\" (max score 100), and another for \"acc-plus\" (max score 100, whether both questions for the same image are correct). The total MME score is the sum of two, so the maximum possible score is 200. Thus, **there is no need to normalize the score to 100 on this benchmark.**\n\n[a] Chaoyou Fu, et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023.\n\n---\n> **`Q5: Dose the classical data augmentation methods have Unexpected Competitiveness?`**\n\nR5: Thank you for your comment. In this work, we evaluated two classical data augmentation methods: crop and rotate. For each method, we averaged the results across different benchmark. The performance gains (Δ) over the baseline are summarized below:\n|Model|COCO-QA-spatial|Spatial-mm|MME-pos|MM-vet-spat|COCO-QA|MMB|VQA_v2|MM-Vet-total|Avg|Spatial_avg|\n|----|---|---|---|---|---|---|---|---|---|---|\n|$\\mathbf{LLaVA_{Rotate}}$Δ|0.55%|2.36%|0.83%|10.86%|1.19%|0.20%|1.85%|3.50%|2.67%|3.65%|\n|$\\mathbf{LLaVA_{Crop}}$Δ|0.15%|1.59%|-0.04%|7.58%|0.58%|-0.18%|1.24%|1.03%|1.49%|2.32%|\n|**$\\mathbf{\\mathbf{LLaVA_{SiTe}}}$Δ**|**1.52%**|**4.19%**|**4.30%**|**7.20%**|**1.45%**|**0.08%**|**1.27%**|**0.93%**|**2.62%**|**4.30%**|\n|$\\mathbf{LLaVA_{Rotate}^{flickr}}$Δ   |-1.35%|1.77%|0.34%|-0.46%|-0.88%|-0.35%|0.04%|0.20%|-0.09%|0.08%|\n|$\\mathbf{LLaVA_{Crop}^{flickr}}$Δ|0.55%|-3.04%|1.64%|10.59%|0.52%|0.96%|0.48%|1.12%|1.60%|2.43%|\n|**$\\mathbf{LLaVA_{SiTe}^{flickr}}$Δ**|**3.57%**|**2.13%**|**1.32%**|**0.85%**|**2.77%**|**0.17%**|**0.76%**|**0.17%**|**1.47%**|**1.97%**|\n|$\\mathbf{Qwen2_{Rotate}}$Δ|-2.44%|-0.17%|-0.41%|-5.31%|-6.95%|-1.66%|8.01%|-6.72%|-1.96%|-2.08%|\n|$\\mathbf{Qwen2_{Crop}}$Δ|-0.31%|-0.42%|0.82%|0.14%|0.03%|-0.71%|9.37%|0.18%|1.14%|0.06%|\n|**$\\mathbf{Qwen2_{SiTe}}$Δ**|**0.51%**|**0.98%**|**3.70%**|**10.52%**|**0.83%**|**0.97%**|**8.12%**|**-1.69%**|**2.99%**|**3.93%**|\n|$\\mathbf{Qwen2_{Rotate}^{flickr}}$Δ|-14.90%|0.51%|13.81%|-13.24%|-11.38%|-7.88%|-5.97%|-12.38%|-6.43%|-3.45%|\n|$\\mathbf{Qwen2_{Crop}^{flickr}}$Δ|-10.04%|1.21%|25.37%|-24.20%|-6.87%|-4.65%|-8.78%|-4.76%|-4.09%|-1.92%|\n|**$\\mathbf{Qwen2_{SiTe}^{flickr}}$Δ**|**1.30%**|**1.16%**|**20.33%**|**13.88%**|**1.63%**|**1.35%**|**0.84%**|**2.86%**|**5.42%**|**9.17%**|\n\n**As shown above, classical methods offer limited gains and often introduce factual conflicts between images and captions, leading to performance drops on some benchmarks (e.g., Spatial-MM, MME_Pos, COCO-QA_Spat). In contrast, SiTe consistently achieves stronger and stable improvements on spatial tasks.**\n\n---\n> **`Q6: Can you explain the effectiveness and necessity of SFT in the overall training pipeline?`**\n\nR6: We are sorry for the confusion. To validate the effectiveness of SiTe, we conducted data augmentation separately in the pretraining and SFT stages, and the the SFT results are **not** further fine-tuning from the pretrained models. Thus, the two stages are not directly comparable.\n\nTo further explore the effectiveness of applying SiTe in both pretraining and SFT stages, we conducted the following experiment:\n\n||COCO-QA-spatial|Spatial-mm|MME-pos|MM-vet-spat|COCO-QA|MMB|VQA_v2|MM-Vet-total|\n|---|---|---|---|---|----|---|---|---|\n|Pretrain(SiTe default)|68.75%|43.78%|133.33|**28.43%**|71.54%|74.43%|60.53%|31.40%|\n|SFT(+1K SiTe)|68.81%|43.16%|128.70|27.06%|71.74%|74.60%|**61.17%**|31.20%|\n|Pretrain+SFT|**69.26%**|**44.38%**| **140.00**|26.62%|**72.11%**|**74.67%**|61.07%|**32.62%**|\n\n**As shown, applying SiTe-based augmentation in both stages yields further improvements across multiple benchmarks, so these stage are both effective and necessary.**\n\n---\nWe hope the above response can further address the concerns and misunderstanding. Thank you again for your feedback!"}, {"review_id": "aWEfHUQcZG", "reviewer": "Reviewer_eNJr", "summary": "The paper presents \"Stitch and Tell (SiTe)\", a novel multimodal data augmentation method designed to enhance the spatial understanding capabilities of vision-language models. These models often struggle with spatial hallucinations, generating incorrect descriptions of object positions in images due to the limited spatial information in existing training data. SiTe addresses this by introducing structured spatial supervision into the training process without requiring costly advanced models or human annotations.", "questions": "please see weakness", "limitations": "The paper does not discuss the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. SiTe presents a novel approach to injecting spatial structure into vision-language training data through image stitching and spatially-aware caption generation.\n2. The idea of using weakly supervised spatial question-answer pairs generated from stitched images is interesting. This approach provides an effective way to introduce spatial supervision without requiring manual annotations or expensive computational resources.\n3. The paper provides extensive experimental results, demonstrating consistent improvements in spatial understanding tasks.\nWeakness:\n1. The complexity of the methodology and the absence of a dedicated limitations section affect clarity. \n2. SiTe involves generating new image-text pairs through stitching and caption generation, which could lead to a significant increase in the size of the training dataset. This might pose scalability challenges when applied to extremely large datasets.\n3. The method relies on the assumption that stitching two images and creating corresponding captions can effectively teach the model spatial understanding. However, this might be an oversimplification of how spatial reasoning is developed in humans and may not be sufficient to capture all nuances of spatial relationships in images.", "was_revised": true, "final_justification": "I incline to accept this paper. The authors address my concerns well in the rebuttal phase.", "rebuttal": "**Dear Reviewer eNJr, we are thrilled that you found the idea is interesting and provides an effective way to introduce spatial supervision without requiring manual annotations and expensive computational resources. We are especially pleased that you recognize our extensive experimental results. Your acknowledgment of the SiTe consistent improvements in spatial understanding tasks is greatly appreciated. We will address your concerns and questions below.**\n\n---\n> **`Q1: Can you provide the complexity of the methodology?`**\n\nR1: SiTe is designed to be both computationally and operationally. The core augmentation procedure consists of two simple steps:\n\n**Image Stitching**: This step involves copying and pasting two images onto a new canvas. Its computational complexity is **linear** in the number of pixels **(O(H*W))**.\n\n**Template-based Caption and QA Generation**: Both operations are either **constant (O(1))** or **linear** in the length of the caption (O(L), where L is the number of words).\n\nIf SiTe is implemented as an online augmentation, stitched images and captions are generated in memory on-the-fly for each training batch, eliminating the need to store them on disk. This results in negligible storage overhead, with space complexity reduced to O(B × H × W × C), where B is the batch size. Only the original dataset needs to be stored. We will clarify this in the revision.\n\n---\n> **`Q2: Where is the limitations section?`**\n\nR2: We would like to clarify that the limitations of our approach are explicitly discussed in **Appendix A**. In this section, we note that the current work focuses on horizontal (left–right) and vertical (top–down) spatial configurations. We also discuss that future work may explore more comprehensive spatial relations by incorporating depth maps, multi-view images, and other settings that enable 3D perception.\n\n---\n> **`Q3: Does this method lead to a significant increase in the size of the training dataset?`**\n\nR3: Thank you for raising this concern. In fact, with SiTe, the total number of training samples is **reduced** rather than increased. For each stitched sample, the two original image-caption pairs are removed from the raw dataset and replaced by a single new sample. As a result, the final training set contains **fewer samples** than the original dataset. This design ensures that SiTe does not pose scalability issues, even for large-scale applications. Furthermore, SiTe can be implemented as an online augmentation, generating stitched samples dynamically during training without increasing storage requirements. We will clarify this point in the revised manuscript.\n\n**In summary, SiTe is both storage-efficient and scalable, reducing the size of the training data and making the overall pipeline more efficient.**\n\n---\n> **`Q4: This method may not be sufficient to capture all nuances of spatial relationships in images`**\n\nR4: Thank you for raising this important point. We agree that our approach is a simplified model of spatial reasoning and does not attempt to replicate all the complexities of human perception. The main goal of SiTe is to inject clear and explicit spatial supervision, which is often lacking in existing datasets. \n\nTo explore the generalization of our method, we selected out-of-distribution spatial relation samples from COCOQA (COCOQA-ood), which include spatial relation keywords such as \"behind\", \"in front of\", \"close to\", \"far away from\", \"inside\", \"outside\", \"within\", \"internal\", \"external\", \"ahead of\", and \"adjacent to\". We further evaluated our models on these as well as on the CV-Bench[a], which specifically test for complex spatial reasoning including depth and occlusion, and on RealWorldQA[b], which is designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The results are shown in the table below:\n\n| Model   | COCOQA-ood | CV-Bench 2d | CV-Bench 3d |RealWorldQA|\n|-----------------|------------|-------------|-------------|---|\n| $\\mathbf{LLaVA_{Baseline}}$  | 70.38%  | 52.56% | 33.43%|55.12%|\n| $\\mathbf{LLaVA_{SiTe}}$  | **71.47%** | **54.28%**  | **35.50%**  |**55.21%**|\n| $\\mathbf{Qwen_{Baseline}}$ | 69.24%  | 46.77%  | 47.63%  | 52.28%|\n| $\\mathbf{Qwen_{SiTe}}$ | **71.34%** | **48.16%**  | **50.36%**  |**54.44%**|\n\nAs shown, the injection of spatial supervision via SiTe leads to improved model performance on out-of-domain spatial relation tasks and demonstrates strong generalization to more complex spatial reasoning scenarios in CV-Bench. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships.\n\nWhile our method focuses on horizontal and vertical relations, we observe in our experiments that the enhanced spatial representations learned through SiTe also generalize to a broader range of spatial reasoning tasks, including cases beyond these basic relations. This demonstrates that SiTe provides a practical and effective foundation for spatial understanding. We believe that SiTe offers a practical and scalable first step, and we explicitly discuss in the limitations section that extending to more complex and realistic spatial relations is an important direction for future work.\n\n**Overall, although SiTe only augments basic directional relations, it demonstrates strong generalization to more complex spatial relationships in practice.**\n\n[a] Tong, Peter, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" NeurIPS 2024.\n\n[b] X.AI. Realworldqa. Blog post, July 2025. Accessed: 2025-07-30.\n\n---\n> **`Q5: The caption of Figure 1: spatially-aware->Spatially-aware.`**\n\nR5: Thanks for pointing this out. We have double-checked and polished our manuscript to fix the typos.\n\n---\nWe hope this response clarifies the remaining concerns and highlights the contributions of our work. Thank you again for your positive comment."}, {"review_id": "Wg1gfhgJxT", "reviewer": "Reviewer_htxQ", "summary": "The paper proposes SiTe, a data augmentation method to enhance the spatial understanding capabilities of MLLMs. The approach consists of two key steps: (1) stitching two images along a spatial axis to create a combined image, and (2) generating spatially grounded question-answer pairs based on the new layout. Experiments on baseline models demonstrate improvements in spatial perception tasks.", "questions": "- How many total samples were used for training. Do you use 1K/5K augmented data along with original data?\n- What is the distribution of different stitched image types and question types?\n- Would the method benefit from testing on SOTA MLLMs?\n- Is the proposed method able to handle more complex reasoning about occlusion, depth, and partial visibility?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed method is simple yet effective, requiring minimal modifications to existing pipelines.\n- SiTe can augment existing training datasets without costly manual annotations, making it scalable.\n\nWeaknesses\n- The paper lacks clarity on dataset details, such as total training size, composition, and diversity.\n- Experiments are conducted on baseline models including LLaVA and HALVA. Validation on state-of-the-art MLLMs (e.g., Qwen2-VL, Intern-VL) is missing. It’s unclear if the proposed method can achieve consistent improvement on these stronger models.\n- The proposed stitching paradigm generates highly synthetic images (e.g., rigid left-right concatenation) that may not reflect the complexity of natural spatial relationships. Real-world scenes often involve occlusion and are influenced by depth and 3D layout. Therefore, while the method improves performance on simplified spatial QA tasks, its generalization to real-world scenarios is limited since spatial understanding requires inferring latent relationships (e.g.,  to answer the question \"Is the cup behind the book?\").", "was_revised": true, "final_justification": "After the rebuttal, I remain concerned about the inherent limitations of SiTe, a binary (\"yes\" or \"no\") answer format, which may restrict its generalization to more complex spatial reasoning tasks. This simplicity could explain the observed performance drop in general vision-language tasks, as also noted by other reviewers. After carefully considering the rebuttal and other reviews, I still find the practicality of SiTe augmentation limited in its current form. Given these unresolved issues, I would like to maintain my original score as borderline reject.", "rebuttal": "**Dear Reviewer htxQ, we thank for your careful and helpful comments on our work. We appreciate the reviewer’s recognition of the simplicity and effectiveness of our approach, as well as the scalability benefits of SiTe. Below are our responses to all the concerns you raised, which will be incorporated into the updated version to further enhance the quality of our submission.**\n\n---\n> **`Q1: How many total samples were used for training? Do you use 1K/5K augmented data along with original data? What is the total training size, composition, and diversity?`**\n\nR1：Thank you for your question! SiTe applies data augmentation in both the pretraining and fine-tuning stages. \n\nIn the pretraining stage, we experimented with two datasets. The first is blip_laion_cc_sbu_558K: $\\mathbf{LLaVA_{Baseline}}$ was trained on 558K image–caption pairs, while $\\mathbf{LLaVA_{SiTe}}$ used 458K pairs, since one-third of the data was stitched and duplicate pairs were removed (as shown in **Table 3** of the manuscript). The second dataset is flickr-30K: $\\mathbf{LLaVA^{Flickr}}$ was trained on 30K image–caption pairs, while $\\mathbf{LLaVA_{SiTe}^{Flickr}}$ used 24K pairs, with one-third of the data being stitched and deduplicated.\n\nThe number of data in pretraining stage is shown in the figure below (taking LLaVA as an example):\n| | $\\mathbf{LLaVA_{Baseline}}$ | $\\mathbf{LLaVA_{SiTe}}$ | $\\mathbf{LLaVA^{Flickr}}$ | $\\mathbf{LLaVA_{SiTe}^{Flickr}}$ |\n|---|----|----|----|-----|\n| **Num of Pretrain Data** | 558K  | 458K (2/3 Raw + 1/3 SiTe)| 30K | 24K (2/3 Raw + 1/3 SiTe) |\n| **Num of SFT Data**  | 665K  | 665K | 665K   | 665K  |\n\nIn the supervised fine-tuning stage, we performed partial stitching on the original llava_v1_5_mix665k dataset. Specifically, we randomly sampled 1K or 5K augmented samples and added them to the original fine-tuning dataset llava_v1_5_mix665k(665K) for full-parameter fine-tuning. This design helps maintain model generalization and avoid overfitting.\n\nThe number of data in supervised fine-tuning stage is shown in the figure below :\n\n| | $\\mathbf{LLaVA_{SiTe}^{1K}}$ | $\\mathbf{LLaVA_{SiTe}^{5K}}$ |\n|-------|-----|-----|\n| **Pretrain Data**| 558K  | 558K  |\n| **SFT Data** | 665K + 1K (SiTe)  | 665K + 5K (SiTe) |\n\nRegarding template diversity, SiTe employs a variety of directional stitching templates, as detailed in Appendix Tables 5, 6, 7, and 8. During augmentation, the stitching format is randomly sampled from these templates, so each template is selected with equal probability. As for the diversity of the training set, since SiTe does not introduce any additional supervision beyond the templates, **the diversity of the augmented data remains consistent with that of the original dataset.**\n\n---\n> **`Q2: What is the distribution of different stitched image types and question types?`**\n\nR2：Thank you for your question. As stated in the paper, the proportion of horizontally and vertically stitched images, as well as question types, is set to 1:1 by design:\n\n*We apply both horizontal and vertical stitching in a 1:1 ratio.*\n\nAnd we specify the default SiTe setting:\n\n*By default, the stitched data make up one-third of the total set.*\n\nThese details can be found in Section 4.2 (Pretraining Stage) of our paper.\n\n---\n> **`Q3: Would the method benefit from testing on SOTA MLLMs?`**\n\nR3：Thank you for your suggestion! Given the architectural similarities between QwenVL and InternVL, and considering that we have already included experiments with LLaVA-Qwen2 in the main text, we further conducted preliminary experiments on InternVL2-4B. We modified the training process to follow a Pretrain + SFT paradigm and applied SiTe augmentation during the pretraining stage. Specifically, we used both the blip_laion_cc_sbu_558K dataset and its SiTe-augmented version (using the default setting).\n\n|Model|COCO-QA-spatial|Spatial-mm|MME-pos|MM-vet-spat|COCO-QA|MMB|VQA_v2|MM-Vet-total|\n|---|----|---|---|---|---|---|---|---|\n|$\\mathbf{InternVL2_{Baseline}}$|51.01%|40.51%|**136.00**|24.1%|55.69%|66.57%|55.10%|24.4%|\n|$\\mathbf{InternVL2_{SiTe}}$|**52.03%**|**41.75%**|129.00|**29.3%**|**56.25%**|**68.35%**|**55.30%**|**25.6%**|\n\nAs shown, InternVL-SiTe consistently outperforms the baseline InternVL on most spatial understanding tasks, while also maintaining strong generalization on general benchmarks. **This demonstrates that SiTe can also benefit SOTA MLLMs.** These results will be incorporated into the updated version to further enhance the quality of our submission.\n\n---\n> **`Q4: Is the proposed method able to handle more complex reasoning about occlusion, depth, and partial visibility?`**\n\nR4: Thank you for your insightful comment! We acknowledge that the current stitching paradigm does not fully capture the complexity of natural spatial relationships. However, SiTe is designed to provide a lightweight and effective way to inject explicit spatial supervision at minimal cost, without requiring complex data collection or annotation.\n\nTo evaluate SiTe's generalization, we selected out-of-distribution spatial relation samples from COCOQA (COCOQA-ood), which include spatial relation keywords such as \"behind\", \"in front of\", \"close to\", \"far away from\", \"inside\", \"outside\", \"within\", \"internal\", \"external\", \"ahead of\", and \"adjacent to\". We further evaluated our models on CV-Bench[a], which specifically tests complex spatial reasoning, including depth and occlusion, and on RealWorldQA[b], which is designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The results are shown in the table below:\n\n| Model   | COCOQA-ood | CV-Bench 2d | CV-Bench 3d |RealWorldQA|\n|-----------------|------------|-------------|-------------|---|\n| $\\mathbf{LLaVA_{Baseline}}$  | 70.38%  | 52.56% | 33.43%|55.12%|\n| $\\mathbf{LLaVA_{SiTe}}$  | **71.47%** | **54.28%**  | **35.50%**  |**55.21%**|\n| $\\mathbf{Qwen_{Baseline}}$ | 69.24%  | 46.77%  | 47.63%  | 52.28%|\n| $\\mathbf{Qwen_{SiTe}}$ | **71.34%** | **48.16%**  | **50.36%**  |**54.44%**|\n\nAs shown, the injection of spatial supervision via SiTe leads to improved model performance on out-of-domain spatial relation tasks and demonstrates strong generalization to more complex spatial reasoning scenarios in CV-Bench and RealWorldQA. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships. We agree that more realistic and comprehensive spatial augmentation is an important direction for future work.\n\n**In summary, although SiTe does not directly address occlusion or 3D reasoning, our results show that it improves generalization to these more complex spatial tasks.**\n\n[a] Tong, Peter, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" NeurIPS 2024.\n\n[b] X.AI. Realworldqa. Blog post, July 2025. Accessed: 2025-07-30\n\n---\nWe thank the reviewer again for these careful comments and hope these responses address your concerns!"}, {"review_id": "aMW0LRZ2BC", "reviewer": "Reviewer_3Q8z", "summary": "This paper introduces \"Stitch and Tell\" (`SiTe` in short), a simple data augmentation strategy for multimodal data that aims at improving spatial understanding of Large Multimodal Models (LMMs). \n\nSiTe works in two steps:  \n1. It stitches two randomly sampled images onto the vertical or horizontal axis (randomly selected);  \n2. It constructs a spatially-aware caption following a template (for example: \"To the left is {caption1}, while the right part contains {caption2}\"). \n\nSiTe also has a version for Instruction-Tuning data, where noun phrases are extracted from the texts and templates constitute spatially-aware questions, such as \"Is {object in left image} to the left of {object in right image}\"?\n\nIn this work, the first variant is used as a data augmentation strategy for pretraining (on image-caption pairs from llava-595k), while the second variant is used for instruction tuning (llava-665k), and evaluation is conducted on both (i) benchmarks probing spatial understanding and (ii) general multimodal benchmarks. \n\nThe paper experiments with LLaVA-v1.5-7B, LLaVA-Qwen2-7B and HALVA, showing that SiTe is effective for both Pretraining (PT) and SFT while not harming performance on standard multimodal benchmarks (e.g., MMBench).", "questions": "For a productive rebuttal, I would discuss the weaknesses above as well as the following questions:\n\n1. A by-product of SiTe is that the average image resolution processed by the LMM is higher. Did the authors evaluate if, as a side effect, also high-resolution image understanding of LMMs trained with SiTe data is improved? I think this would be a nice finding if true; \n2. The image stitching algorithm (Algorithm 1) states that images are pasted on a blank canvas, which is sized according to the larger vertical / horizontal edge between the two sampled images. This inevitably leads to areas of the canvas which will remain blank, which are possibly very large if the sampled images have large differences in size, and consequently entails throughput inefficiencies since there will be many unneeded visual tokens. Naive solutions to this would be proportional resizing of the image with the smaller edge, or sampling / grouping images of similar size. Did the authors consider these aspects?\n3. The experimental section says that 5 runs were executed. Could the authors please report the standard deviation as well? I think this would be important to understand if the improvements provided by SiTe are significant.", "limitations": "The paper contains a limitations section in the appendix (which mentions 3D spatial understanding). I think this is sufficient, even though ideally it should fit the main body of the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths.**\n1. This paper introduces a simple idea, which I believe we should promote and not penalize;  \n2. Spatial Understanding is overall an important problem to tackle, as it serves many applications (*e.g.*, robotics or autonomous driving); \n3. SiTe is presented in two variants (PT and SFT), showing effectiveness in both scenarios;  \n4. SiTe is used to augment only a subset of PT or SFT data, showing that spatial understanding can be boosted without compromising general multimodal knowledge;  \n5. By reducing the overall number of data points fairly, SiTe also incurs a lower computational budget than standard SFT since the overall number of data points is reduced (stitched images are not re-used data points); \n\n\n**Weaknesses.**  \n1. The related work of this paper introduces other multimodal data-augmentation strategies. I understand some of these may be heavier compared to SiTe, but I wonder why none of them are empirically validated in this work (even though they are not \"spatially-aware\", showing they don't translate into any downstream gain for spatial understanding might make the paper more complete);  \n2. Given the nature of flat image stitching (e.g., images are stitched on a flat 2D canvas), I wonder whether SiTe hurts or benefits spatial understanding when it comes to a notion of depth (so, spatial relationships as \"{obj1} is behind of {obj2}\"). I think it would be nice to also evaluate on CV-Bench [a] (both the 2D and 3D subsets), which has gained quite some popularity in the multimodal community over the past year. The 3D subset would specifically serve as a tool to probe if SiTe would unlock \"depth-aware\" spatial understanding as a by-product.\n\n**References.**  \n[a] Tong, Peter, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" NeurIPS 2024.", "was_revised": true, "final_justification": "The discussion with the authors was positive, introducing results for 3D spatial understanding, exploring more efficient ways to stitch images together, and examining high-resolution image understanding as a by-product of the proposed method. My score was positive from the beginning, and I am keeping it.", "rebuttal": "**Dear Reviewer 3Q8z, we truly appreciate your thorough review, positive feedback, and the time and effort you devoted to strengthening our paper. We are grateful that you highlighted the simplicity and practical value of our approach, as well as the importance of spatial understanding for real-world applications. We are also encouraged by your recognition of the effectiveness of SiTe. Your thoughtful remarks are highly motivating for us.**\n\n---\n> **`Q1:Did the authors evaluate if, as a side effect, also high-resolution image understanding of LMMs trained with SiTe data is improved?`**\n\nR1: Thanks and we really like this insightful comment! We conducted additional experiments on the HR-Bench 8K [a] and V-Star [b] high-resolution datasets. **As shown in the table below, models enhanced with SiTe consistently achieve better performance on these high-resolution benchmarks.** This result motivates us to further explore the potential of SiTe for more aspects in visual understanding task!\n\n|Model|HR-Bench|V*|\n|----------------|---------|------|\n|$\\mathbf{LLaVA_{Baseline}}$|30.90|48.34|\n|$\\mathbf{LLaVA_{SiTe}}$|**32.56**|**49.95**|\n|$\\mathbf{Qwen_{Baseline}}$|32.25|37.83|\n|$\\mathbf{Qwen_{SiTe}}$|**32.67**|**38.92**|\n\n[a] Wang, Wenbin, et al. \"Divide, conquer  and combine: A training-free framework for high-resolution image  perception in multimodal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.\n\n[b] Wu, Penghao, and Saining Xie. \"V*: Guided visual search as a core mechanism in multimodal llms.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n\n---\n> **`Q2: Did the authors consider the resizing or grouping the images of similar size?`**\n\nR2：Thank you for your thoughtful suggestion! We found your idea very interesting and have incorporated it by improving the SiTe stitching strategy.\n\n$\\mathbf{SiTe_{resize}}$: For left-right stitching, we resize the shorter image so that both images have the same height; for top-down stitching, we resize the narrower image so that both images have the same width.\n\n$\\mathbf{SiTe_{ratio}}$: We first categorize images based on their aspect ratios. Images with width/height > 1.2 are grouped for top-down stitching, and images with height/width > 1.2 are grouped for left-right stitching.\n\n$\\mathbf{SiTe_{default}}$ refers to the default random stitching method described in the paper.\n\nWe conducted experiments using these variants in the pretraining stage of LLaVA with the blip_laion_cc_sbu_558K(558K) dataset. The results are shown below:\n\n|Model|COCO-QA-spatial|Spatial-MM|MME-pos|MM-vet-spat|COCO-QA|MMB|VQA_v2|MM-Vet-total|\n|---|---|---|---|---|----|---|---|---|\n|$\\mathbf{Baseline}$|67.72|42.02|127.83|26.52|70.52|73.50|60.48|31.11|\n|$\\mathbf{Site_{resize}}$|68.19|42.98|130.25|27.97|70.94|73.24|60.52|31.46|\n|$\\mathbf{Site_{ratio}}$|**70.26**|**44.15**|132.80|**28.68**|71.19|73.64|**60.57**|**32.27**|\n|$\\mathbf{Site_{default}}$|68.75|43.78|**133.33**|28.43|**71.54**|**74.43**|60.53|31.40|\n\nAs shown, $\\mathbf{SiTe_{ratio}}$ achieves better on most benchmarks. We suppose this is because grouping images by ratio ensures that the stitched images have more realistic proportions, avoiding unnatural shapes that are too wide or too tall. $\\mathbf{SiTe_{resize}}$ may create stitched images with resolutions that differ from the testset, which could negatively affect performance.\n\n---\n> **`Q3: Did SiTe hurt or benefits spatial understanding when it comes to a notion of depth?`**\n\nR3：Thank you for raising this concern! We addressed it by further evaluating our models on CV-Bench[a] and on RealWorldQA[b], which focuses on basic real-world spatial understanding in multimodal models. The results are presented below.\n|Model|CV-Bench 2d|CV-Bench 3d|RealWorldQA|\n|---|---|---|---|\n|$\\mathbf{LLaVA_{Baseline}}$|52.56|33.43|55.12|\n|$\\mathbf{LLaVA_{SiTe}}$|**54.28**|**35.50**|**55.21**|\n|$\\mathbf{Qwen_{Baseline}}$ |46.77|47.63|52.28|\n|$\\mathbf{Qwen_{SiTe}}$ |**48.16**|**50.36**|**54.44**|\n\nAs shown, **the injection of spatial supervision via SiTe leads to improved model performance on out-of-domain spatial relation tasks and demonstrates strong generalization to more complex spatial reasoning scenarios in CV-Bench and RealWorldQA**. A reasonable hypothesis is that the model benefits from its improved understanding of fundamental directional relationships.\n\n[a] Tong, Peter, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" NeurIPS 2024.\n\n[b] X.AI. Realworldqa. Blog post, July 2025. Accessed: 2025-07-30.\n\n---\n> **`Q4：Could the authors please report the standard deviation as well?`**\n\nR4: Of course! We compute the standard deviation in the below table. Due to space limitations, we show a subset of the results here.\n\n|Model|COCO-QA_Spat|Spatial-MM|MME_Pos|MM-Vet_Spat|COCO-QA|VQAv2|MMB|MM-Vet|\n|---|---|---|---|---|---|---|---|---|\n|$\\mathbf{LLaVA_{Baseline}}$|67.72±0.10|42.02±0.54|127.83±1.07|26.52±0.08|70.52±0.32|60.48±0.16|73.50±0.14|31.11±0.19|\n|$\\mathbf{LLaVA_{Rotate}}$|68.09±0.17|43.01±0.11|128.89±1.51|29.40±0.03|71.36±0.11|60.60±0.17|74.86±0.05|32.20±0.12|\n|$\\mathbf{LLaVA_{Crop}}$|67.82±0.25|42.69±0.24|127.78±0.94|28.53±0.12|70.93±0.18|60.37±0.24|74.41±0.09|31.43±0.25|\n|$\\mathbf{LLaVA_{VSR}}$|67.85±0.21|42.65±0.36|121.67±1.01|29.42±0.54|70.90±0.28|60.57±0.25|73.88±0.14|31.47±0.07|\n|$\\mathbf{LLaVA_{SiTe}}$|68.75±0.06|43.78±0.14|133.33±0.43|28.43±0.19|71.54±0.27|60.53±0.23|74.43±0.03|31.40±0.04|\n|$\\mathbf{LLaVA_{SiTe}^{1K}}$|68.81±0.26|43.16±0.27|128.70±0.08|27.06±0.11|71.74±0.47|61.17±0.29|74.60±0.26|31.20±0.03|\n|$\\mathbf{LLaVA_{SiTe}^{5K}}$|67.75±0.04|46.21±0.14|139.26±0.31|28.10±0.23|70.96±0.13|60.38±0.20|74.53±0.18|30.69±0.24|\n|$\\mathbf{Qwen2_{SiTe}^{1K}}$|58.53±0.47|41.24±0.03|65.33±0.54|23.10±0.44|61.77±1.32|54.58±0.03|66.37±0.11|23.93±0.22|\n|$\\mathbf{Qwen2_{SiTe}^{5K}}$|59.95±0.53|42.22±0.21|61.67±0.53|23.20±0.37|63.00±1.10|54.75±0.14|66.43±0.13|23.93±0.19|\n\n**As indicated by the standard deviation results, the performance gains are consistent and robust**, and we will add the whole results in the revised paper.\n\n---\n> **`Q5：Are you considering comparing other data augmentation methods, even they are more heavier?`**\n\nR5：Thanks for the suggestion! Following approaches similar to [a] and [b], we performed data augmentation by using a multimodal large language model to rewrite the original captions and inject spatial information into the training data.\nWe used the following prompt for GPT-4o to rewrite captions based on the 558K.\n>Based on the image and its original caption, Your task is to generate a new caption that prioritizes spatial relationships, no more than 20 words.\n\nTo evaluate the rewritten captions[c], we used Qwen2.5VL-7B as a judge with the following prompt:\n>Your task is to compare two captions for an image and determine which better meets the evaluation criteria. (Print ONLY 1, -1, or 0 as the result)\n>Evaluation Criteria:\n>1. Accuracy: How accurately the caption describes the actual content of the image.\n>2. Positional Details: The quantity and quality of positional/directional descriptions.\n>Result: Return 1 if Caption1 is better, -1 if Caption2 is better, and 0 if equally good.\n>Caption1: {new_caption}\n>Caption2: {old_caption}\n\nThe results are as follows:\n```\nScore=1: 94.36%, Score=0: 2.97%, Score=-1: 2.68%\n```\n\nThese results confirm that the rewritten captions significantly improve spatial semantic richness and descriptive accuracy.\n\nWe then used the rewritten captions to train LLaVA in the pretraining stage. The results are shown below:\n\n|Model|COCO-QA-spatial|Spatial-MM|MME-pos|MM-vet-spat|COCO-QA|MMB|VQA_v2|MM-Vet-total|\n|----|---|---|---|---|---|---|---|---|\n|$\\mathbf{LLaVA_{Baseline}}$|67.72|42.02|127.83|26.52|70.52|73.50|60.48|31.11|\n|$\\mathbf{LLaVA_{Rewrite}}$|**70.39**|**44.95**|129.50|**30.40**|**73.01**|73.34|**61.92**|**32.15**|\n|$\\mathbf{LLaVA_{SiTe}}$|68.75|43.78|**133.33**|28.43|71.54|**74.43**|60.53|31.40|\n\nAs shown, rewriting captions leads to significant performance improvements, demonstrating the importance of explicitly insert spatial information in captions. This approach also produces captions that are more coherent and natural, injecting richer supervisory signals and more detailed information.\n\nHowever, it is worth noting that caption rewriting for the 558K dataset cost approximately $560 using the GPT-4o API, and this expense would increase significantly for even larger datasets (not including the additional cost of caption quality evaluation).\n\nAs described in our paper, SiTe focuses on providing a lightweight, effective, and unambiguous way to inject spatial supervision. In future work, we plan to explore combining SiTe with caption rewriting to further enhance spatial semantic richness in the training data.\n\n[a] Chen G H, et al. Allava: Harnessing gpt4v-synthesized data for lite vision-language models[J]. arXiv preprint arXiv:2402.11684, 2024.\n\n[b] Chen L, et al. Sharegpt4v: Improving large multi-modal models with better captions[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 370-387.\n\n[c] Lee, Yebin, et al. Fleur: An explainable reference-free evaluation metric for image  captioning using a large multimodal model. arXiv preprint arXiv:2406.06004 (2024).\n\n---\nWe thank the review again for this **insightful** and **interesting** comment. We will include the above analysis in the revised paper to make the paper more complete. Thanks again!"}]

Author rebuttal:

Meta-review: ### Summary

The paper tackles the spatial hallucination issues in multimodal LLMs. The authors propose SiTe, which creates new training examples by stitching two images and generating new spatial-aware captions. The authors claim that augmenting training data with SiTe enhances a model's spatial understanding without requiring architectural changes or expensive manual labeling. Their experiments across multiple model architectures (LLaVA, Qwen2) show that SiTe significantly improves performance on dedicated spatial reasoning benchmarks (e.g., MME-Position, Spatial-MM) while maintaining or even slightly improving performance on general vision-language benchmarks.

### Strengths

- Simplicity and practicality: The proposed method is simple, elegant, and easy to implement. As noted by all reviewers (3L35, eNJr, htxQ, 3Q8z), its plug-and-play nature, requiring no changes to model architecture, makes it a highly practical and valuable contribution to the community.
- Addresses a significant problem: The paper is well-motivated, tackling the important and widely recognized problem of spatial reasoning in multimodal models.
- Effectiveness and efficiency: The experimental results compellingly demonstrate that this simple idea works. SiTe consistently improves spatial understanding across various models and datasets. 

### Weaknesses
The initial reviews identified several potential weaknesses, most of which were effectively resolved during the rebuttal period. 

- Scalability during supervised fine-tuning (SFT): The primary remaining concern, raised by Reviewer htxQ, involves the method's behavior during SFT. The initial experiments showed that scaling up the number of SiTe-augmented samples from 1K to 5K could degrade performance on general benchmarks.
- Limited scope: The original paper focused primarily on basic 2D spatial relationships (left/right, above/below). While the authors demonstrated impressive generalization to more complex and 3D relations during the rebuttal, an analysis of why this generalization occurs was not part of the original submission and remains an interesting avenue for future work (noted by Reviewer 3L35).

### Recommendation and Justification

I am recommending an Accept for this paper. The core idea is simple, intuitive, and demonstrably effective at addressing a well-known problem. While not a complete solution to all types of spatial reasoning, it represents a clear, practical, and useful step forward. The most important reasons for this recommendation are:
- The paper offers a practical, easy-to-use tool that other researchers can immediately apply to improve their models. This kind of straightforward and effective contribution is highly valuable to the community.
- The authors' response to the reviews was effective. The rebuttal and discussion provided substantial new experimental evidence that directly addressed the reviewers' concerns about generalization, baseline comparisons, and performance on state-of-the-art models. 
- Positive reviewer consensus: Following the rebuttal, three of the four reviewers converged on a positive recommendation (two weak accepts, one accept). The remaining reviewer's concerns about SFT scalability were valid but were, in my view, effectively mitigated by the authors' new experiments showing that the issue could be resolved with more diverse templates—a strength of the framework's flexibility.

### Summary of Discussion and Rebuttal

The authors provided a rather strong rebuttal, conducting a significant number of new experiments that addressed nearly every concern raised by the reviewers, thereby substantially strengthening the paper. The following concerns have been discussed. 

- Generalization to complex relations: Reviewers (3L35, htxQ, 3Q8z) questioned whether the method would generalize beyond simple left/right and above/below relations to more complex 3D and depth-aware scenarios. The authors conducted new experiments on the CV-Bench (2D and 3D) and RealWorldQA benchmarks, demonstrating that SiTe indeed improves performance on these more complex tasks.
- Comparison with simpler baselines: Reviewer 3L35 initially felt the gains over standard augmentations like Rotate and Crop were marginal. The authors provided comprehensive tables showing that, unlike SiTe, the baseline augmentations were unstable and sometimes harmed performance. They further provided results on CV-Bench showing SiTe's superiority.
- Scalability of SFT: Reviewer htxQ raised a critical point about performance degradation on general benchmarks when scaling SiTe data during SFT. The authors conducted new SFT experiments by introducing 42 diverse QA templates (including multiple-choice, open-ended, and fill-in-the-blank) and a more robust ratio-based image stitching strategy, they showed significant gains on spatial benchmarks without sacrificing performance on general benchmarks.
-Performance on SOTA models: Reviewer htxQ questioned the method's effectiveness on stronger models. The authors ran new tests on InternVL, showing consistent improvements.
Decision: Accept (poster)