SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with Layout Generation Methods. We also conduct experiments to compare with layout generation methods and the corresponding results are presented in Table 6. The experimental results include FID and Align. score (referred from LayoutDM [13]) with lower values indicates better performance. Notably, VISORGPT significantly surpasses these state-of-the-art methods with better FID and Align.
Table 6: Comparison to previous methods.
Methods	Rico	PubLayNet
FID (↓)	Align. (↓)	FID (↓)	Align. (↓)
MaskGIT [2]	52.1	0.015	27.1	0.101
BLT [16]	88.2	1.030	116	0.153
BART [19]	11.9	0.090	16.6	0.116
VQDiffusion [9]	7.46	0.178	15.4	0.193
LayoutDM [13]	6.65	0.162	13.9	0.195
VISORGPT (Ours)	5.85	0.109	9.18	0.103
score on Rico and PubLayNet datasets, showcasing the superior capability of VISORGPT to model and generate layouts.
4.3 Visualization Results
[bottle, dining table, person, knife, bowl, bowl, oven, person, cup, cup, bowl, bowl, broccoli, spoon]
potted plant, chair, chair, dining table, refrigerator, banana, oven, sink, orange, orange, orange, orange, chair, orange
Figure 3: Generation diversity of VISORGPT.
Generation Diversity. One of the concerns is that VISORGPT would memorize the overall data distribution and cannot generate diverse and novel layouts. We present examples in Fig. 3 to address the concern. Firstly, we specify target classes (e.g., "bottle, dining table, person, knife, bowl, oven, person, cup, cup, bowl, bowl, broccoli, spoon") and then select layouts satisfying these conditions from COCO train set (left in Fig. 3, only one satisfied sample over 110K samples). In contrast, we use the same target classes as prompts for VISORGPT to infer their bounding boxes. As depicted in Fig. 3, the bounding boxes generated by VISORGPT exhibit diversity and considerable differences from those selected ones from COCO train set.
Scene Completion. VISORGPT can receive user conditions and reasonably complete the corresponding missing classes in an interactive way, which can also demonstrate the capability of novel/unseen layout generation. We present examples in Fig. 4. One can observe that users can roughly draw two instances of "person" across the canvas (above) and prompt VISORGPT to deduce the remaining "skis" (below). To avoid cherry-picking, we provide the distribution of generated "skis" using 100 samples when given various scenes.
Figure 4: Scene completion.
As illustrated, when the two "person" vary from left-top to middle-bottom, the distribution of "skis"
Figure 6: Shape prior of the categories of 'person', 'car', 'motorcycle', and 'teddy bear'.
exhibits consistent variations. This validates that VIsORGPT does not rely on memorization. Instead, it learns intrinsic visual priors among categories, empowering it to infer novel/unseen layouts, accommodating the diverse requirements of users.
Relation Prior. Fig. 5 illustrates the comparison between the real-world relation matrix among 30 categories and the one estimated by VISORGPT. Each row depicts the relation prior of one category to others. For instance, it can be observed from the real world matrix that the 'person' (the first row) frequently interacts with other categories such as 'dog' and 'cat'. Similarly, in the
Figure 5: Relation matrix among 30 categories on COCO.
third row, the co-occurrence between 'car' and 'bus', 'truck', and 'stop sign' is larger than that of other categories. Notably, it is clear that the relation prior learned by VISORGPT is very close to that of the real-world one. This indicates that VISORGPT can capture the real relationships among categories and generate sequential output that aligns with these visual prior.
Location Prior. In addition to the quantitative results presented above, we visualize the comparison between the location prior learned by VISORGPT and the real one across various categories. Fig. 7 displays the location prior of three categories, including 'surfboard', 'tie', and 'train'. It is noticeable that, in each column, the location prior learned by VISORGPT is similar to the real one. For instance, from the first column, one can observe that the real distribution of 'tie' is mainly located in the lower-middle region, and the shape prior learned by VISORGPT exhibits a similar pattern.
Table 7: Location prior of some categories.
Shape Prior. Fig. 6 shows the shape prior of four categories, such as 'person' and 'motorcycle'. To facilitate comparison, we employ kernel density estimation to estimate a continuous distribution from the discrete one. We observe that the shape prior learned by VISORGPT is close to those of the real visual world. For example, in the real world, the ratio of width to height of a car is almost always larger than 1, and the estimated shape prior of 'car' is mainly distributed around 1.8. It is evident that the learned probabilistic prior by VISORGPT, represented by the blue line, closely approximates the real one, represented by the red line. Overall, the shape priors of other categories learned by VISORGPT well match that of the real world.
Tab. 8 presents the impact of Special Words (SW), Textual Knowledge (TK, i.e., with model weights initialized from the official pre-trained GPT-2), the number of sequences (#Seq), and model size (#Param). (a) Results on Tab. 8a are measured by the average KL divergence of location and shape prior. This confirms that the special words can potentially improve VISORGPT's performance in learning the visual prior. Notably, we found that the NLP textual knowledge deteriorated the performance of VISORGPT. We attribute this to the fact that the association between visual coordinates
Table 8: Impact of Special Words (SW), Textual Knowledge (TK), Number of Sequences (#Seq), and Model size (#Param). We use KL Div  $(\downarrow)$  as evaluation metric.
(a) Effect of SW and TK.
SW	TK	COCO	Open Images
✓	×	1.647	1.895
×	×	1.720	1.950
×	✓	1.959	2.240
(b) Effect of #Seq.
#Seq	COCO	Open Images
~40	1.958	3.247
~80	1.195	2.460
~120	0.930	2.144
(c) Effect of #Param.
#Param	COCO
117M	0.850
345M	0.836
762M	0.798
and natural language is relatively weak, thus it becomes inessential to learn visual prior from visual annotations. (b) In Tab. 8b, we find that increasing the number of sampled sequences leads to a more precise estimation of the visual prior by VISORGPT. (c) In Tab. 8c, we investigate the impact of model size on learning visual prior. For simplicity and efficiency, we replace VISORGPT architecture by three GPT versions and train it using only COCO (box) data. The results demonstrate the scalability of VISORGPT, i.e., modeling the visual prior better with increased learnable parameters.
Conditions from VISORGPT
Synthetic Images
Conditions not obeying prior
Figure 7: Comparison of synthetic images from object boxes adhering to the prior (left) or not (right).
4.5 Applications
Conditional Image Synthesis. VISORGPT's remarkable ability to infer visual categories and their locations based on user-customized prompts shows promising potential for generating customized images that still maintain a sense of realism. Here, we utilize ControlNet [43] and GLIGEN [22] to synthesize images from keypoints and bounding-boxes, respectively. We showcase some examples in Fig. 7 and 8. The first and fourth columns in Fig. 7 present the customized spatial conditions sampled from VISORGPT and the conditions not adhering to the visual prior. The second, third, and fifth columns provide synthetic results by GLIGEN conditioned on the corresponding spatial conditions. For example, on the first three columns, it is evident that the spatial conditions sampled from VISORGPT are more natural, such that the synthetic images are realistic and natural-looking. However, when the conditions (the last two columns) do not adhere to the prior, such as 'person' not being on a similar scale to 'dining table', the width of 'pizza' being too long, and the width of 'chair' being too short, the synthetic contents like 'person', 'chair', and 'dining table' appear abnormal, also impacting the authenticity of other objects like the two cups (circled in red dotted line).
Moreover, VISORGPT is capable of inferring sequences that include instances with keypoint information. For example, as shown in Fig. 8, we can provide a prompt like "key point; multiple instances; large; 13; 18; person," to VISORGPT. This allows it to conditionally imagine a scene involving 13 people with their keypoint coordinates. Decoded results can be used as spatial conditions for image synthesis by ControlNet (shown in the last two columns). More examples can be found in supp.
This work proposed a novel approach, VISORGPT, to explicitly learning the probabilistic prior of the visual world through generative pre-training. This was achieved by transforming the continuous visual locations into discrete tokens by prompting and training a transformer decoder to maximize
Input prompt:
key point; multiple instances; large; 13; 18; person,
Output sequence:
key point; multiple instances; large; 13; 18; person, person, person, person, person, person, person, person, person, person, person, person, person, person; [a: 252 158 b 260 182 c 276 184 d 280 222 e 0 0 44 181 g 0 0 0 265 247 i 0 0 0 1 234 m 0 0 n 0 0 0 259 154 p 250 153 a 269 156 r o ] [a: 172 163 b 174 199 c 200 198 d 212 236 e 206 279 f 147 199 g 135 238 h 132 284 i 195 200 j 320 381 l 158 280 m 0 134 b 160 166 h 160 q 186 l 159 r [6] [a: 1356 b 374 b 200 t c 396 198 d 412 245 v t 347 240 h 344 z 282 i t 395 284 j t 403 345 k t 412 405 t 1364 b 285 m t 371 t347 h t380 o t 361 h o t 376 s r [o ] [a: 302 h t o b   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   .t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t   .t     ] [a: 195 b 276 m o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o     ] [a: 92 177 b 93 b 203 c 116 b 204 d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d e d.
key point; multiple instances; large; 3; 18; person,
key point; multiple instances; large; 3; 18; person, person, person; [a 364 210 b 401 258 c 357 258 d 320 327 e 290 290 f 446 258 g 436 333 h 388 331 348 267 f 343 269 i 0 408 376 m 0 0 n 0 0 0 0 364 203 p 373 203 q 0 0 r 404 207 ] [a 169 203 b 170 248 c 121 248 d 81 344 e 131 279 f 221 248 g 254 327 h 192 257 i 128 385 j 64 396 k 0 0 n 215 388 m 215 399 n 0 0 155 192 p 184 195 q 136 207 r 196 209 ] [a 256 188 b 277 c 235 237 z 232 d 0 0 e 0 136 z 237 g 330 307 h 262 z6 363 j 0 0 k 0 01 301 366 m o n o 0 0 0 254 r 180 267 f 182 q o r 287 193 ]
key point; multiple instances; large; 4; 18; person,
key point; multiple instances; large; 4; 18; person, person, person, person; [a: 0 0 b 252 87 c 504 258 d 508 281 e 0 0 f 0 0 g 0 0 h 0 0 i 509 302 j 509 340 k 508 372 i 0 0 m 0 0 0 o 0 0 p q r o r o] [a: 412 227 b 420 265 c 445 267 d 464 313 e 435 329 f 395 263 g 360 301 a 407 298 i 436 372 j 0 0 k 0 01 398 370 m 0 0 o 0 242 k 421 p 403 q 220 p 439 z r 396 z zj ] [a: 254 167 b 233 197 c 272 181 i 329 176 e 374 171 f 194 b13 g 184 z67 h 161 331 z632 r155 b285 j 396 k0 f 1229 320 m 241 387 n 0 0 o 254 160 p243 z63 o q z244 z69 ] [a: 0 b0.10 w78.490 z175.500 z686 e0 f0 g0 h0 o i496 305 j501.377 z0 k10 o o1o o o o o o o o o o o o o
Figure 8: Illustration of input prompts (comprising multiple instances with keypoints), output sequences, decoded results and synthetic images.
the likelihood of training sequences. As a result, VISORGPT exhibits significant potential in comprehending real-world visual prior and leveraging this knowledge to create plausible scenes under a variety of customized prompts. This ability can facilitate the automatic synthesis of a vast number of images, along with their corresponding fine-grained annotations, using ControlNet and GLIGEN. This could potentially yield ample resources to train more robust visual intelligence models.
Broader Impact. This work demonstrates the substantial capacity of Large Language Models (LLMs) to effectively model spatial visual locations through language modeling. In the subsequent generations of LLMs, the visual priors discussed here can be seamlessly integrated into large-scale training, endowing LLMs with the ability to possess not only textual knowledge but also the capability to perceive and understand the visual world. While current datasets may have limited annotations for object categories, the vast knowledge present in extensive text corpora allows for the association of known objects with novel ones through language-based knowledge, such as synonyms. This presents a promising direction for the next generation of LLMs to expand their understanding and modeling of our visual world.
**Limitation.** We encountered limitations regarding the number of instances that could be included in each sequence due to the maximum token length, despite converting each mask annotation to a fixed length. In the future, we plan to incorporate large-scale natural language corpora into training and extend the maximum sequence length.
This project is supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF13-2021-0008, and the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2022).
6.1 Advantages compared to selecting GT bounding boxes from train set
(i) Diverse layouts generation. When users are required to specify a large number of classes (larger than 10), the layouts selected from COCO train set including all such classes are limited. As shown in Fig. 3, when target classes ('bottle', 'dining table', 'person', 'knife', 'bowl', 'bowl', 'oven', 'person', 'cup', 'cup', 'bowl', 'bowl', 'broccoli', 'spoon') are considered, there is only one relevant layout containing all the target classes from COCO train set. In contrast, as shown in Fig. 3, VISORGPT can receive these target classes to infer a lot of corresponding layouts with much diversity.
(ii) User-customized / Controllability. Compared to selecting from ground truth bounding boxes, one of the main advantages of VISORGPT lies in its ability to extrapolate novel layouts based on users' configurations, as depicted in Fig. 10 (c). VISORGPT can deduce the missing classes' positions ('skis, skis, skis, skis') based on the layout ('person, person') given by users.
(iii) Free-form captions as layout condition. Incorporating captions in the training sequences enables VISORGPT to receive free-form natural language input as a condition (e.g., a boy and a girl are playing by the sea or a train is running toward the man) and generate corresponding layouts (as shown in Fig. 10 (b)).
6.2 Extrapolation ability
To showcase VISORGPT's extrapolation ability, we present examples in Fig. 10 (a). As demonstrated, when the 'sea' occupies the majority of the canvas, the 'surfboard' can appear throughout the entire canvas. Upon shifting the 'sea' region to the middle part of the canvas, the 'surfboard' is accordingly confined within the middle region. This phenomenon demonstrates that VISORGPT does not solely rely on memorization; rather, it learns intrinsic relation, location, and shape priors among categories. Consequently, VISORGPT can extrapolate unseen/novel layouts based on users' configurations.
6.3 Examples of Training Sequences
Here, we give some examples of various types of training sequences on different datasets:
Human Pose (COCO):
key point; multiple instances; large; 1; 18; person; [ a 190 120 b 266 146 c 318 143 d 385 232 e 338 269 f 214 150 g 0 0 h 0 0 i 312 280 j 365 296 k 359 420 l 258 283 m 194 344 n 301 383 o 197 100 p 181 103 q 234 84 r 0 0]
Human Pose (CrowdPose):
key point; multiple instances; large; 2; 14; person, person; [ a 312 201 b 306 200 c 311 232 d 269 214 e 298 257 f 231 206 g 296 275 h 307 275 i 251 244 j 271 235 k 274 292 l 283 295 m 304 153 n 310 191 ] [ a 179 247 b 165 245 c 164 313 d 160 315 e 221 316 f 207 279 g 155 343 h 144 366 i 242 337 j 240 367 k 210 431 l 300 418 m 172 176 n 177 227] key point; multiple instances; large; 2; 14; person, person; [ a 240 178 b 304 168 c 228 239 d 0 0 e 261 236 f 0 0 g 251 296 h 289 296 i 0 0 j 0 0 k 0 010 0 m 261 92 n 272 156 ] [ a 314 160 b 363 158 c 274 232 d 356 264 e 224 260 f 271 263 g 298 315 h 341 324 i 0 0 j 332 442 k 0 010 0 m 287 64 n 333 133]
Instance Mask:
mask; multiple instances; medium; 1; 0; clock; [ m0 224 291 m1 226 299 m2 227 306 m3 228 313 m4 233 320 m5 238 325 m6 245 329 m7 252 332 m8 259 334 m9 266 335 m10 274 333 m11 281 330 m12 288 327 m13 293 323 m14 299 318 m15 303 312 m16 305 305 m17 307 298 m18 310 291 m19 308 284 m20 307 276 m21 303 269 m22 299 263 m23 295 257 m24 288 254 m25 280 251 m26 273 250 m27 266 249 m28 259 249 m29 252 251 m30 246 256 m31 240 260 m32 235 265 m33 229 270 m34 227 277 m35 225 284]
Object Centric Bounding-Box:
box; object centric; large; 1; 0; castle; [ xmin 236 ymin 142 xmax 413 ymax 232]
6.4 Implementation Details
All experimental evaluations were conducted on eight NVIDIA Tesla V100-32GB GPUs using PyTorch. In order to include special words, we created a new vocabulary containing a total of 30,769 words based on a standard vocabulary. To optimize computational efficiency and memory utilization, we utilized the DeepSpeed framework. To serialize visual locations, we first resized the long side of each image to a length of 512 pixels
and then shifted the image content to the center by padding the short side to a length of 512 pixels. As a result, the number of bins  $m$  was set to 512. The flag of [Size] indicates the average area of all instances in the image and we set the flag according to the rule:
$$
\left\{ \begin{array}{c c} \text {` ` s m a l l ` '} & \text {a v e r a g e a r e a <   3 2 ^ {2}} \\ \text {` ` m e d i u m ` '} & 3 2 ^ {2} \leq \text {a v e r a g e a r e a <   9 6 ^ {2}} \\ \text {` ` l a r g e ` '} & \text {a v e r a g e a r e a \geq 9 6 ^ {2}} \end{array} \right..
$$
We omitted person instances with fewer than five keypoints. To enable continuous generation, we designed and trained models based on the prompt format (b). Specifically, VISORGPT $^{\dagger}$  (a&b) and VISORGPT (a&b) were trained using the same number of sequences as VISORGPT $^{\dagger}$  (a) and VISORGPT (a), respectively. The only difference is that we randomly utilized prompt format (a) or (b) to construct each training sequence.
During the evaluation stage, we set the maximum sequence length of our model (VISORGPT) to 256 tokens to ensure efficient inference. In the ablation studies, we added special words only to the [Coordinate] term, and we reported the average KL divergence between the location and shape priors learned by VISORGPT and those in the real world. Since training large-scale language models is time- and resource-consuming, we trained only three types of VISORGPT with respect to GPT-2 (base, medium, large) with a maximum token length of 256 in 50,000 iterations on COCO (Box) data.
6.5 Evaluation Details
To estimate discrete visual prior from VISORGPT, we infer a series of sequences via prompting as below:
Code in Python:  
f"box; multiple instances; random.choice(['small', 'medium', 'large'])); random.randint(2, 10); 0; category name,"
To ensure that each category in a given dataset is sufficiently represented in the sequence data used for estimating the visual prior, we specify a minimum number of sequences in which each category must appear. Table 9 provides an overview of the predicted sequences that are used for evaluation.
Table 9: Details about the predicted sequences for evaluation.
Datasets	#Categories	#Predicted Seq.	Min #Seq. Per Category
Open Images (Box)	600	48,000	~80
Objects365 (Box)	365	29,200	~80
COCO (Box)	80	6,400	~80
In our study, we adopt the Kullback-Leibler divergence to quantify the similarity between two given discrete distributions. Specifically, let  $p$  and  $q$  denote the estimated probabilistic priors derived from the real-world data and the VISORGPT, respectively. The degree of similarity between these two distributions can be computed as:
$$
\operatorname {K L} (p \| q) = p \log (p / q). \tag {2}
$$
6.6 Visualization
Relation Prior of COCO. Fig. 9 illustrates the comparison between the real and learned relation prior among 80 categories on the COCO dataset. As can be observed, there is a high degree of similarity between the two relation matrices.
More Visual Comparison. We provide more comparison of visual prior between the real world and one learned by our VISORGPT and failure cases on COCO dataset in Fig. 11.
Continuous Generation. Fig. 12 presents a set of examples showcasing continuous generation based on the current scene. Notably, in each row, the proposed VISORGPT is able to successfully complete a scene that involves many individuals annotated with 14/18 keypoints or objects with bounding boxes, based on the information provided in the corresponding scene depicted in the previous columns.
Figs. 13 and 14 present more visualization results.
Figure 9: Relation among 80 categories on COCO.
(a) Prior of 'surfboard' adapts to variations of the 'sea'
(b) Supports to receive natural language as input
(d) Multimodal inference of boxes and keypoints
(c) Scene completion based on user provided objects
Figure 10: More visual examples.
(a) Comparison of location prior
(b) Comparison of shape prior
(b) Failure cases
Figure 11: Comparison of visual prior between the real world and one learned by VISORGPT on COCO dataset.
(a) Continuous generation on COCO (keypoint)
(b) Continuous generation on CrowdPose
(c) Continuous generation on COCO (Box)
Figure 12: Examples of continual generation.
Continuous Generation based on the current scene
key point; multiple instances; large; 13; 18; person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person.
key point; multiple instances; large; 8; 18; person,
key point : multiple instances ; large; 8 ; 18 ; person, person, person, person, person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person , person .
key point; multiple instances; large; 8; 18; person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person, person.
key point; multiple instances; large; 5; 18; person,