Learning Visual Prior via Generative Pre-Training
Authors: Jinheng Xie
Kai Ye
Yudong Li
Yuexiang Li
Kevin Qinghong Lin
Yefeng Zheng
Linlin Shen
Mike Zheng Shou
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Prior
Generative Pre-Training
Conditional Image Synthesis

Abstract: Various stuff and things in visual data possess specific traits, which can be learned by deep neural networks and are implicitly represented as the visual prior, e.g., object location and shape, in the model. Such prior potentially impacts many vision tasks. For example, in conditional image synthesis, spatial conditions failing to adhere to the prior can result in visually inaccurate synthetic results. This work aims to explicitly learn the visual prior and enable the customization of sampling. Inspired by advances in language modeling, we propose to learn Visual prior via Generative Pre-Training, dubbed VisorGPT. By discretizing visual locations, e.g., bounding boxes, human pose, and instance masks, into sequences, VisorGPT can model visual prior through likelihood maximization. Besides, prompt engineering is investigated to unify various visual locations and enable customized sampling of sequential outputs from the learned prior. Experimental results demonstrate the effectiveness of VisorGPT in modeling visual prior and extrapolating to novel scenes, potentially motivating that discrete visual locations can be integrated into the learning paradigm of current language models to further perceive visual world. Code is available at https://sierkinhane.github.io/visor-gpt.

Learning Visual Prior via Generative Pre- Pre-Training
Jinheng Xie'
Kai Ye
Yudong Li Yuexiang Li
Kevin Qinghong Lin'
Yefeng Zheng² Linlin Shen Mike Zheng Shou
Show Lab, National University of Singapore 2 Shenzhen University
Jarvis Research Center, Tencent YouTu Lab
(sierkinhane, mi ke. zheng. shou) (aw.oul.)mmmm...oom.
https://sierkinnie wtw.aaaaa....tttii.iooo...o
Abstract
Various stuff and things in visual data possess specific traits, which can be learned by deep neural networks and are implicitly represented as the visual prior, e.g. object location and shape, the model. Such prior potentially impacts many vision tasks. For example, in conditional image synthesis, spatial conditions failing to adhere to the prior can result in visually inaccurate synthetic results. This work aims to explicitly learn the visual prior and enable the customization of sampling. Inspired by advances in language modeling, we propose to learn Visual prior via Generative Pre- Tre-Training dubbed VISORGPT. By discretizing visual locations, e,g., bounding boxes, human pose, and instance masks, into sequences, VISORGPT can model visual prior through likelihood maximization. Besides, prompt engineering is investigated to unify various visual locations and enable customized sampling of sequential outputs from the learned prior. Experimental results demonstrate the effectiveness of VISORGPT in modeling visual prior and extrapolating to novel scenes, potentially motivating that discrete visual locations can be integrated into the learning paradigm of current language models to further perceive visual world.
Introduction
The digital camera can continuously capture photographs of the visual world, such that tremendous photos and videos are currently shared on the Internet. In our world, various stuff and things possess specific traits, which have been correspondingly embedded in such visual data. In the current era of deep learning, deep neural networks [10, 40, 7] have demonstrated remarkable proficiency in learning from vast amounts of data, leading to the development of visual foundation models (VFMs) [15, 32, 43, 12, 22, 20]. Such traity have been accordingly learned and implicitly represented as the visual prior in VFMs, which has the potential to impact real-world applications. An example that highlights its importance can be seen in the field of image synthesis. To present high-quality and natural-looking images, the synthetic stuff and things must adhere to the visual prior such as the spatial location, shape, and interaction of objects (Fig. I (a)). A vivid example of layout-to-image provided in Fig. 1 (b). When the spatial conditions do not adhere to the visual prior, such as the shape of 'donut' not being square, the size of 'person' being similar to that of 'donut', and 'donut' being floated in the air instead of being placed on 'dining table', the resulting synthetic contents may be inaccurate and visually inconsistent with the desired outcome. Despite recent advances in conditional image synthesis such as ControlNet [43] and GLIGEN [22], the challenge of continuously sampling customized spatial conditions that adhere the visual prior remains difficult problem, particularly for automatic synthesis of massive images with corresponding fine-grained annotations.
In this paper, study the problem of how to explicitly learn visual prior from the real world and enable customization of sampling. If we would like to paint a series of instances on canvas, we
Equal Contribution
Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Synthesize
Synthesize
(a) Visual Prior
(b) Conditions not adhering the prior
(c) Conditions adhering the prior
Object Location
Shaaee and Relation, dic
Visual World
Sequence corpus
box;
multiple instances;
large; 6; 0; cup, donut,
multiple instances; large; 6; O; cup, donut, donut, donut, donut, dining table; xmin 37 ymin 136 xmax 183
box;
VISORGPT
(d) Learning Visual Prior via Generative Pre- Pre-Training (e) Customized sampling from VISoRGPT Figure I: An overview of the problem of visual prior (top) and VISORGPT (bottom). (a) refers to visual prior, e-g,, location, shape, and relations of objects. (b) provides a failure case of image synthesis from spatial conditions that do not adhere to the prior. Specifically, the shape of the 'donut' being square and 'donut' being floated in the air instead of being placed on 'dining table'. (c) displays success case that conditions sampled from VISORGPT leads to more accurate synthetic results. (d) illustrates that VISORGPT learns visual prior through sequence corpus converted from the visual world. (e) gives an example that a user customizes a sampling from VISORGPT by prompting. should decide what to paint and also their shapes, locations, interactions, etc. It seems that these elements share joint probabilistic prior, in which any stuff or things can be accordingly sampled construct scene. As there may be many potential variables in the prior, is extremely hard to be comprehensively formulated. Over the past few years, significant advances have been made in language modeling [28, 29, 1, 6, 46], demonstrating their remarkable capacity for modeling the probabilistic distribution of sentences. Our focus is on learning the visual prior of location, shape, and relationships among categories, rather than raw pixels. It possible to convert such visual information into a series of sequences, such that the visual prior can be learned by language modeling. To this end. as presented in Fig. 1 (d), we propose to learn Visual prior via Generative Pre- Pre-Training. dubbed VISORGPT, Thanks to the development of deep learning, many high-qualite annotated data such as bounding-box [23, 38, 17]. human pose [23, 21]. instance mask [23] are publicly available. This provides sufficient location, shape, and relation information of stuff and things in the visual world. Since they are all encoded using 2D or 3D coordinates, we can simply convert them into a corpus of sequences. In this way, the visual prior can be learned by pretext objective, e.g., maximizing the likelihood each sequence. Beyond this, prompt engineering is investigated to unify various visual locations and enable the customized sampling of sequential outputs from the learned prior.
As shown in Fig. (e), according the user's prompt, VISORGPT can correspondingly sample sequence from the learned prior, which can be spatially decoded for image synthesis (Fig. (c)). Since the decoded conditions adhere the prior, the synthetic 'cup'. 'dining table', and 'donut' are realistic and consistent with the desired semantics. This finding confirms that we can continuously customize spatial conditions from many aspects, e.g., data type, object size, number instances, and classes, using VISORGPT. With the advance of conditional image synthesis, it is feasible to generate an endless supply of synthetic images with their corresponding fine- ien-grained annotations, potentially providing ample resources to train more robust and generalized visual intelligence models.
Related Works
Language Modeling. Language modeling aims to estimate the probability of given sequence of words occurring in a sentence. In recent years, Large language models (LLMs) such as GPT series [28, 29, 1] and BERT family [6, 18, 26] have revolutionized the field of natural language processing. In particular, BERT family adopts the encoder(-decoder) architecture and employs masked language modeling techniques to model each given sentence bi-directionally in context. In contrast, GPT series employs the decoder-only architecture to sequentially model the probability of

---

next tokens by maximizing the likelihood of given sentences. Such pretext objective allows for easy scaling up in terms of the model's parameters and training corpus. In this work, inspired by GPT series, we investigate the potential of decoder-only architecture in modeling visual priors.
Layout Generation. Several related works [45, 4, 16, 2, 13] have focused scene graphs layout generation, specifically in the context of mobile interfaces and documents, For instance, Inoue er al. [13] addressed structural data using discrete diffusion models and provided a unified approach to diverse layout generation tasks. More recently, Feng et al. [8] directly delved into layout generation using LLMs via in-context learning. In contrast to these methods, our approach goes beyond the layout of interfaces and documents. We extend our focus to encompass various modalities, including human keypoints and semantic masks, and unify the modeling of this diverse information within single straightforward learning objective.
Conditional Image Synthesis. With large-scale image-text datasets [36, 35), generative mod- els [31, 30, 34, 32, 25, 24], e.g., DALL-E, Imagen, and Stable Diffusion, have shown a significant capacity for synthesizing images higher quality and greater diversity. Recently, more controllable image synthesis models such as training-bases ControlNet [43] and GLIGEN [22] and training free BoxDiff [42], have demonstrated a remarkable ability to control the synthetic contents precisely. When it comes to generating an extensive set of novel images, relying solely on spatial conditions from users or referring from images is inefficient. To tackle this problem, our VISORGPT is capable of continuously sampling customized and novel spatial conditions, making it possible to synthesize endless streams of data for various practical applications.
3 Methodology
In this section, we begin by presenting our problem formulation (5 3.1) and prompt designs (8 3.2) for unifying various visual information (e.g., class and location) as textual sequences. Building upon this, introduce model architecture and pretext objective 3.3) model the visual prior. Finally, provide practical examples of how to sample customized sequential outputs from VISORGPT ($ 3.4).
3.1 Problem Formulation
We assume that visual location x, e.g., object bounding-box, human pose, and instance mask, follow a probabilistic prior distribution Pæ. However, since pæ is often unavailable in practice, this work aims learn model fe with parameters θ that can empirically approximate the latent probabilistic prior Pa- By doing so, we can sample new instances z from the learned prior, denoted as æ ~ fe. to facilitate various vision tasks, such as conditional image synthesis and action generation.
3.2 Visual Location as Sequences
In the discrete language domain, we witness that variety of tasks, e.g., translation and question- answering, can be integrated as a unified template (e.g. prompts and responses) and then processed by one language model using different user prompts. However, when it comes to annotations of visual locations such as 2D object bounding-box instance mask, and 3D human pose which are continuous and highly structured, unified approach has yet to be explored and our objective is to investigate potential solutions to this issue. Following Chen et al. [3], we discretize visual annotations, i.e., continuous numbers, into m bins, such that location information can also be naturally represented as discrete tokens and m integers are then accordingly added to the standard vocabulary. It means that coordinates can be represented by a sequence of words. In particular, each number representing visual localization will be quantified as an integer in the range of [1, m). In this way, visual locations of each image be then unified into a sequence t PROMPT (x))
As visual annotations of various tasks in different formats, we propose three universal prompts:
Prompt template Ta:
Annotation type; Data type; Size; #lnstances; #Keypoints; Category names; Coordinates
Prompt template Ty:
Annotation type; Data type; Size; #Instances; #Keypoints; [Category name Coordinate i];
Prompt template Te:
Annotation type; Natural Language Input: Category names; Coordinates; Instruction; Category names;
Coordinates
3

---

The provided prompts can be summarized in Tab. 1. which provides standardized templates unify com- monly used 2D and 3D visual location information sequences sequences. Each prompt begins with the flags [Annotation type] and [Data type], which
Table Candidate choices prompt template.
Annotation type Data type
box; keypoint; mask; multimodal; object centric; multiple instances small; medium: large
into ID textual
#lnstances
I; 2: 3;
#Keypoints
14:18
are the flags indicating the type of annotation and scene, e.g., box and multiple instances. The follow- ing flags of [Size] and [#Instance]] represent the average area and the number of instances in the current image, while [#Keypoins]] indicates the number of keypoints annotated for each person, i.e., 14 or 18. The following two flags are the [Category name] of each instance and their correspond- ing [Coordinate]. At last, [Natural Language Input] is extended for free-form language input and [Instruction] is used for iterative refinement of the current layout. Examples of these prompts are presented in the following sections. By employing our defined templates, transform commonly used visual annotations into a large-scale sequential corpus. The corpus can be seamlessly ingested by language models, facilitating better learning of visual commonsense prior.
Calegury name
cups person; dog:
3.3 Learning Visual Prior via Generative Pre-Training
Model Architecture. In the past few years, many large language models have been successively proposed, such GPT [28, 29, 1] and BERT [6, 18, 26] family, and recently introduced LLaMA [39]. We employ the GPT decoder-style transformer as our model to learn the visual probabilistic prior.
Pretext Objective. After processing the visual locations x: as textual sequences t in 5 3.2, we tokenize each sequence by byte-pair encoding (BPE) algorithm [37] to obtain a sequence with n tokens u = {u1, 122.... : un } such that a standard language modeling objective can be directly employed to learn visual prior by maximizing the following likelihood:
L Σ logp((u ( 1;-1; O),
where k is the size of context window, and p(-I-) indicates the conditional probability which is modeled by the neural network O. Stochastic gradient descent used train the neural network.
3.4 Customizing Sequential Output
In addition to offering formatted visual annotations for learning a probabilistic prior, the standardized templates enable personalize sequential output for various applications through prompting. For example, the customized sequential output can be employed as spatial conditions in image synthesis models (e.g., ControlNet [43] and GLIGEN [22]). This opens up the possibility of synthesizing broad range of data types to address diverse problems and challenges in computer vision. Here are a few representative scenarios:
(a) Object Bounding- Box. As we use a flag to distinguish different types of visual annotations, we can control the type of data and scene to be sampled from the learned probabilistic prior by setting the beginning tokens in the input prompt. Accordingly, we can set the beginning prompt as "box;" to generate sequential output with instances and corresponding bounding-box information. Besides, with flags like [Size], [#Instances] and [#Keypoints], we can sample a scene that adheres to multiple conditions. As depicted in Fig. 2 (a), we can input prompt "box; multiple instances; small; 16; O; kite, kite, person," as a prefix to require the VISORGPT to conditionally infer the remaining tokens. In this example, VISORGPT outputs the categories and their locations, specifically fulfilling the requirement of objects being in small size. In particular, (xmin, ymin) and (xmax, ymax) are special tokens indicating the lop-left and bottom-right corners of the target object.
(b) Human Pose. With flags of [#Instances] and [#Kyypontts], VISORGPT is capable of customizing sequential outputs involving instances with keypoints in a crowd scene. We give an example in Fig. 2 (b). Numbers (10 and 14) are added to the beginning of prompt as conditions to infer a scene consisting of 10 people with 14 keypoints. Note that, use "a, b, c, d, .... and "m0, ml, m2, m3, as special tokens to distinguish each human keypoint and object boundary coordinate, respectively.
(c) Instance Mask. Beyond sparse coordinates as shown in (a) and (b), VISORGPT can deal with dense spatial annotations, i.e. instance masks. Typically, pixel-level information can be represented

---

Customizing Sequential Outputs
Decoded Results
ww..:........................... ................................
(a)
Object Box
box; multiple instances; small; 16; kite, kite, person,
box; multiple instances; small; 16; kite, kite, person, kite, kite, person, person, kite, kite, person, kite, person, kite, kite, person, person; [ xmin 307 ymin 200 xmax 321 ymax 209 ] xmin 208 ymin 235 xmax 208 ymax 238 xmin 200 ymin 286 xmax 204 ymax 295
(b) Human Pose
key point; multiple instances; medium; 10; 14;
key point; multiple instances; medium; 10; 14; person, person, person, person, person, person, person, person, person, person; 196 196 178 196 202 210 172 212 204 224 f 174 225 193 233 h 179 232 0 0 100.000000000 184 174 186 190
(c) Instance Mask
mask; multiple instances; large; 3; 0; person, car,
mask; multiple instances; large; 3; 0; person, car, bus; m0 483 295 m1 483 296 m2 483 297 m3 483 299 m4 483 300 m5 483 302 m6 483 306 m7 483 312 m8 484 330 m9 489 330 m10 495 330 m11 495 311 m12 495 305 m13 495 302 m14 495 300 495 298 m16 495 297 m17 495 ...
(d) Multimodal Inference
multimodal; multiple instances; large; 3; 0;
[CLS] multimodal; multiple instances; large; 3; 18; frisbee, person, person; xmin ymin 285 xmax 384 ymax 332 ] xmmin ymin
(e) Natural Language Input
box; a boy and girl are playing by the sea;
[CLS] box: boy and girl playing by the sea boy a girl, the sea xmiin 219 ymin 95 xmax 310 ymax 291)
Figure Examples of customizing sequential outputs from the proposed VISORGPT.
using a mask matrix or a set of boundary coordinates. For convenient sequentialization, we uniformly sample n points along the angle in the polar space from object boundary coordinates to represent the pixel-level location, which is similar to [41]. We provide an example in Fig. 2 (c).
(d) Multimodal Inference. VISORGPT exhibits the capability of multimodal inference. In Fig. (d), VISORGPT can deduce the presence two individuals and a frishee along with their bounding boxes and keypoints of these people are also predicted the same time.
(e) Natural Language Input and Instruction. Apart from the above prompts, VISORGPT can receive free-form language as input to generate corresponding layouts. We present example in Fig. 2 (d). In addition, the proposed VISORGPT supports iterative refinement by instructions like "make the boy and girl closer".
Table 2: Training corpus for VISORGPT.
4 Experiments
Datasets (type)
#Categories
#lmages
Open Images (Box) [17]
600
1,743,042
4.1 Experimental Setup
Objects365 (Box) [38]
365
1,728,775
COCO (Box) [23]
80
117.266
ImageNet (Box) [33]
1,000
38,285
Datasets. We collect around million se- quences from the publicly available datasets for VISORGPT. In particular, we consider three types of commonly used visual annotations, i.e., object bounding-box, human pose, and instance mask. In the MS-COCO dataset [23], we collect
COCO (Keypoint) [23]
53,473
CrowdPose (Keypoint) [21]
9,981
COCO (Mask) [23]
80
117,266
Flickr30K [27]
44,518
31,783
Rico [5]
25
35,851
Publ.y .etNett [44]
315,757
5

---

-118K images annotated with 80 categories and their object counding-boxee and instance masks. For each image, all object bounding- -oxxess and instance masks with their category information are formatted to a sequence, respectively. Beyond that, ~3.5 million hounding-box annotations of Objects365 [38] and Open Images [17] are also converted to sequences. Other types of annotations (ie., human keypoint) of MS-COCO (-54K) and CrowdPose (-10K) are also formatted to sequential data. For the object-centric scenario, we collect -4K sequences from ImageNet- IK [33]. Besides, Flick30K [27] is employed to support free-form natural language as input. For the scenarios of mobile interface and document layout, we use the Rico [5] and PubLayNet [44] datasets. A summary is presented in Tab. 2.
We propose evaluate VISORGPT from three aspects: (i) Evaluating the quality of sequences generated by VISORGPT. In the inference stage, as VISORGPT predicts sequences in the format given in 3.2, it is necessary to examine whether the generated sequences can be decoded into visual locations. In particular, we generate series of sequences using VISORGPT and caleulate the accuracy whether it can be successfully decoded (termed Format in Table 5) and the number of categories matches the number of locations (termed Matching in Table 5). (ii) As discussed in 3.2, we use flags, i.e. [Size] and [#Instances], to indicate the average size and number of instances in the current sequence. Hence, we can control the average object size and the number of instances in the generated sequences via setting flags [Size] and [#Instances]. Then, we can calculate the accuracy whether the object size and the number of instances in the generated sequences are consistent with the given flags to validate the performance of controllability (termed Size and #Instances, respectively). (iii) Evaluating the learned probabilistic prior, i.e., object location, shape, and relation among categories, on the val set of COCO, Objects365, and Open Images datasets. In this work, we propose compare the discrete distribution of every visual prior. Specifically, to compute the location prior a category, we initialize an empty canvas and convert the bounding-box of each instance of the category binary mask. Then, each mask is accumulated on the canvas and normalized as 2D location distribution. To compute the shape prior of a category. we calculate the ratio of width height of each instance of the category, and estimate discrete distribution as the shape prior. To establish the relation prior of a category to other categories, we count the number of co-occurrences between the category and other categories and estimate a discrete distribution. In this way, discrete prior of each category can be computed on COCO, Objects365,, and OpenImages val sets as real one. Durring evaluation, we infer a series of sequences to compute the learned visual prior. Then we measure the similarity between learned and the real prior using the Kullback-Leiilerr is adopted to compare with layout generation methods [16, 13).
Evaluation Metrics.
divergence [14]. In addition, FID [1
Table 3; Model card of VISORGPT.
Models
#Parameters
d'rmining data
Annotation type
Batch size
Iterations
Learning rate
Sequence length
VISORGPT
117M
4M box & keypoint mask
128
200K
5.0e
1024
VISORGPT!
117M
keypoint
128
200K
5.0e
1024
Implementation Details. We provide training details of VISORGPT in Tab. 3. VISORGPT adopted GPT-2 (base) architecture and was trained from scratch. We use all datasets reported in Tab. 2 to train VISORGPT, and Open Images and Objects365 are not involved train VISORGPTI, In evaluation, each category is at least involved in ~80 valid predicted sequences by prompting (8 3.2).
Table 4: Evaluation on training corpus scale and prompt templates of VISORGPT. The similarity between real probabilistic prior and the learned one measured by KL divergence (KL Div).
KL Div on COCO (L)
KL Div on Open Images (L)
KL Div on Objects365 (L)
Models
Prompt
Location
Shape
Relation
Location
Shape
Relation
Location
Shape
Relation
VisORGPTT
Ta
1.133
1.483
0.452
VISORGPTt
T.+To
1.032
1,446
0.445
VISORGPT
Ta
1.212
1.813
0.561
0.890
2.775
3.715 3.888
1.969 1.995
1.345
2.790
VISORGPT
Ta+To
1,583
1.710
0.581
1.007
2.782
1,377
2.765
4.2
Quantitative Results
Table 5: Evaluation on customized outputs (%).
Evaluation on Learned Visual Prior. In Tab. 4,
Quality (1)
Controllability (1)
we present the measured similarity between real probabilistic prior and the one learned by VI- SORGPT on the validation sets of COCO, Open Images, and Objects365, using KI. divergence. The prompt template T, and T2 +T, in & 3.2, are
Datasets
Format
Matching
Size
#Instances
COCO
100.0
100.0
92.02
100.0
Images
99.97
99.40
89.35
98.71
Objects365
99.99
99.94
91.52
99.78

---

used for comparison. Overall, VISORGPT To and Ta+T, exhibit comparable performance, indicating both prompt templates have comparable capability for learning visual prior.
Evaluation on Customized Sequences. We present the quality of generated sequences and the performance of VISORGPT's controllability in Tab. It obvious that nearly all predicted sequences can be decoded successfully in three datasets. Additionally, in over 99% of sequences, all instances can match their respective locations. Besides, the table shows that VISORGPT achieves accuracies of 92.02%, 89.35% %, and 91.52% in controlling the average object size on COCO, Open Images, and 0bjects365 datasets, respectively. Furthermore, VISORGPT can achieve an accuracy of over 98% in controlling the number of instances across all three datasets. These findings demonstrate the strong capacity VISORGPT in reasoning high-quality sequences and control the object size and number of instances in the scene.
Comparison with Layout Generation Meth- ods. We also conduct experiments to com- pare with layout generation methods and the corresponding results are presented in Table 6. The experimental results include FID and Align. score (referred from LayoutDM [13]) with lower values indicates better performance. Notably, VISORGPT significantly surpasses these state- of-the-art methods with better FID and Align. score on Rico and Publ LayNet datasets, showcasing the superior capability of VISORGPT to model and generate layouts.
Table 6: Comparison to previous methods.
Rico
PubLay
Methods
FID (L)
Align. (L)
FID (L)
Align. (4)
MaskGIT
52.1
0.015
0.101
BLT [16]
88.2
1.030
116
0.153
BART
11.9
16.6
0.116
VQDiilusion [9]
7.46
0.178
15.4
0.193
LaysuDMM
6.65
VISORGPT (Ours)
5.85
0.109
9.18
0.103
4.3 Visualization Results
['bottle,
dining cup,
table, cup,
person, kntfe, bowl,
bowl, bowl, oven, ['potted plant, chair, chair, dining table, refrigerator, banana, broccoli,
person,
bowl,
spoon'] oven, sink, orange, orange, orange, orange, chair, orange")
From train set
Generated by VisorGPT
From train set Figure 3: Generation diversity of VISORGPT.
Generated by VisorGPT
Generation Diversity. One of the concerns is that VISORGPT would memorize the overall data distribution and cannot generate diverse and novel layouts. We present examples in Fig. 3 to address the concern. Firstly, we specify target classes (e.g., "bottle, dining table, person, knife, bowl, oven, person, cup, cup, bowl, bowl, broccoli, spoon")) and then select layouts satisfying these conditions from COCO train set (left in Fig. 3, only one satisfied sample over 110K samples). In contrast, we use the same target classes as prompts for VISORGPT to infer their bounding boxes. As depicted in Fig. 3, the bounding boxes generated by VISORGPT exhibit diversity and considerable differences from those selected ones from COCO train set.
Scene Completion. VISORGPT can re- ceive user conditions and reasonably complete the corresponding missing classes in interactive way, which can also demonstrate the capability of novel/unseen layout generation. We
current scene
present examples in Fig. 4. One can ob- serve that users can roughly draw instances of "person" across the canvas (above) and prompt VISORGPT de- duce the remaining "skis" (below). To avoid cherry-pickingg
perribbttion 'skiss
we provide the distribution of generated "skis" using 100 samples when given various scenes.
Figure 4: Scene completion. As illustrated, when the two "person" vary from left-top middle-bottom, the distribution of "skis"

---

Person
Car
Motorcycle
Teddy bear
World
Rew World
Real World MtsoGPT
Real World
Viseroort
ViscrtiPt
333
880
Figure 6: Shape prior of the categories of 'person', 'car', 'motorcycle', and 'teddy bear".
exhibits consistent variations. This validates that VISORGPT does not rely on memorization. In- stead, learns intrinsic visual priors among categories, empowering to infer novel/unseen layouts, accommodating the diverse requirements of users.
Relation Prior. Fig. illustrates the comparison between the real- world relation matrix among 30 categories and the one estimated by VISORGPT. Each row depicts the relation prior of one cate- gory to others. For instance, it can be observed from the real world matrix that the 'person' (the first row) frequently inter- acts with other categories such as 'dog' and 'cat'. Similarly, in the third row, the co-occurrence between 'car' and 'bus', 'truck', and 'stop sign' is larger than that of other categories. Notably, it is clear that the relation prior learned by VISORGPT is very close to that the real-world one. This indicates that VISORGPT can capture the real relationships among categories and generate sequential output that aligns with these visual prior.
Real World
VisorGPT
partig
Figure 5: Relation matrix among 30 categories on COCO.
Location Prior. In addition to the quantitative re- sults presented above, we visualize the compar- ison between the location prior learned by VI- SORGPT and the real one across various categories. Fig. 7 displays the location prior three categories, including 'surfboard', 'tie', and 'train', It is notice- able that, in each column, the location prior learned by VISORGPT is similar to the real one. For in- stance, from the first column, one can observe that the real distribution of 'tie' is mainly located in the lower-middle region, and the shape prior learned by VISORGPT exhibits similar pattern.
Surfboord
Pooa World
ViorGPT
Table 7: Location prior of some categories.
Shape Prior: Fig. 6 shows the shape prior of four categories, such as 'person' and 'motorcycle', To facilitate comparison, we employ kernel density estimation to estimate continuous distribution from the discrete one. We observe that the shape prior learned by VISORGPT is close to those of the real visual world. For example, in the real world, the ratio of width to height of car is almost always larger than 1, and the estimated shape prior of 'car' is mainly distributed around 1.8. It is evident that the learned probabilistic prior by VISORGPT, represented by the blue line, closely approximates the real one, represented by the red line. Overall, the shape priors of other categories learned by VISORGPT well match that of the real world.
4.4 Ablation Studies
Tab. presents the impact of Special Words (SW), Textual Knowledge (TK, i.e., with model weights initialized from the official pre-trained GPT-2), the number of sequences (#Seq), and model size (#Param). (a) Results on Tab. 8a are measured by the average KL divergence of location and shape prior. This confirms that the special words can potentially improve VISORGPT's performance in learning the visual prior. Notably, we found that the NLP textual knowledge deteriorated the perfor- mance of VPT. We attribute this to the fact that the association between visual coordinates

---

Table 8: Impact Special Words (SW), Textual Knowledge (TK), Number of Sequences (#Seq), and Model size (#Param). We use KL Div (4) as evaluation metric.
(a) Effect of SW and TK.
(b) Effect of #Seq.
(c) Effect of #Param.
SW
TK
COCO
Open Images
#Seq
COCO
Open Images
#Param
COCO
1.647
1.895
-40
1.958
3.247
117M
0.850
1.720
1.950
-80
1.195
2.460
345M
0.836
1.959
2.240
-120
0.930
2.144
762M
0.798
and natural language is relatively weak, thus becomes inessential learn visual prior from visual annotations. (b) In Tab. 8b, we find that increasing the number of sampled sequences leads to more precise estimation of the visual prior by VISORGPT. (c) Tab. 8c, we investigate the impact of model size on learning visual prior. For simplicity and efficiency, we replace VISORGPT architecture by three GPT versions and train it using only COCO (box) data. The results demonstrate the scalability of VISORGPT, i.e. modeling the visual prior better with increased learnable parameters.
Conditions from VIDRGPT
Synthetic Images
Conditions not obeying prior
Synthetic Images
NIT
Figure 7: Comparison of synthetic images from object boxes adhering to the prior (left) or not (right).
4.5 Applications
Conditional Image Synthesis. VISORGPT's remarkable ability to infer visual categories and their locations based on user-customized prompts shows promising potential for generating customized images that still maintain sense of realism. Here, we utilize ControlNet [43] and GLIGEN [22] to synthesize images from keypoints and mounding-boxes. respectively. We showcase some examples in Fig. 7 and 8, The first and fourth columns in Fig. 7 present the customized spatial conditions adhering the visual prior. The second, third,
sampled from VISORGPT and the conditions and fifth columns provide synthetic results by GLIGEN conditioned on the corresponding spatial conditions. For example, on the first three columns, it evident that the spatial conditions sampled from VISORGPT are more natural, such that the synthetic images are realistic and natural-looking However, when the conditions (the last two columns) being a similar scale 'dining table', the width of 'pizza' being too long, and the width of 'chair' being too short, the synthetic contents like 'person', 'chair", and 'dining table' appear abnormal, also impacting the authenticity of other objects like the two cups (circled in red dotted line).
not adhere the prior, such as 'person' not
Moreover, VISORGPT is capable inferring sequences that include instances with keypoint informa- tion. For example, shown in Fig. 8, we can provide prompt like "key point; multiple instances; large: 13; 18; person, VISORGPT. This allows conditionally imagine scene involving 13 people with their keypoint coordinates. Decoded results can be used as spatial conditions for image synthesis by ControlNet (shown in the last two columns). More examples can be found supp.
Conclusion
This work proposed a novel approach, VISORGPT, to explicitly learning the probabilistic prior of the visual world through generative pre-training. This was achieved by transforming the continuous visual locations into discrete tokens by prompting and training transformer decoder to maximize

---

PRATIN Posoeees 38: peemoa peItOn SRIVEN JOSTOOLL DETSER DIPORT ENTHAP EROOON UNOWANT ENITEO SRT UNTTER BRIOR 1 2C1 DM 2 2E Th 022 o Cnd pupte
6:2 150 a 18e LGD 166 159 390 a 28G L60 374 300 300 30B 022 DIC A00 303 201
3O 340 320 2001 DIG 412 MSS 3EA 380 IN SIC 250 YOU ORC 130 & ohd 2
334 339
LOO
30
200
148 17N dOO 2 EN 377 8800 223
140
301 140 302 300 124 20 304 234 4C2
D LOB
2373
2246 R22 170
200)
2013 383 224 330 337 333 300+ 202 320
LES 222 340 323 223 272 GOO 232 2213 327 256
243 ai5 3
2440
226
DOG
383 178c 3.0 d 273 278 goo.ooooooooooo 0000000001000 C OO 3-0 2-0
C11 174 DO OO OO OOP-TT ATT JRG 220 AAS
1000 354, 3000000 1552
170 622 482 1700 HBU 320 500 LET SE A LOG 230
ZAR 22H BON SS BON
20422 LOT
201.
pereer,
1o. ligll oun, largu parce, permarn purmony a 364 330 400
NNA
E aio 17t Eitiv di 164 p 975 201 455 201 1f 166 30.8 2400
240 244 3788 321 y 234
3571 120 (4
397 200 (000000 116- g 500 NEP 292 576 2017 56-9 (0:00-000 (811 506
0. Ed0tp 760 9877g D. 201 1991
mubtple betanccec/agge
GRUERR
Ostpal resamee
87 256 281 www.000000000o00oo 00hO0 902 900
372 00000
End e p 90 1 412 227 265
428 366
LGT :97-
LTG
L72 194
235 196 h 183 355 1M6 321 120 3958 0 123 303 201 327
000 250 160
24 119 LD
30 18: 23
LOO1 56 -
Figure 8: Illustration of input prompts (comprising multiple instances with keypoints), output sequences, decoded results and synthetic images.
the likelihood of training sequences. As a result, VISORGPT exhibits significant potential in com- prehending real-world visual prior and leveraging this knowledge to create plausible scenes under a variety of customized prompts. This ability can facilitate the automatic synthesis of a vast number of images, along with their corresponding fine-grained annotations, using ControlNet and GLIGEN. This could potentially yield ample resources to train more robust visual intelligence models.
This work demonstrates the substantial capacity of Large Language Models (LLMs) to effectively model spatial visual locations through language modeling, In the subsequent generations of LLMs, the visual priors discussed here can be seamlessly integrated into large-scall training, endowing LLMs with the ability to possess not only textual knowledge but also the capability to perceive and understand the visual world. While current datasets may have limited annotations for object categories, the vast knowledge present in extensive text corpora allows for the association of known objects with novel ones through language- based knowledge, such as synonyms. This presents a promising direction for the next generation of LLMS to expand their understanding and modeling of our visual world.
Broader Impact.
We encountered limitations regarding the number of instances that could be included in each sequence due to the maximum token length, despite converting each mask annotation to fixed length. In the future, we plan to incorporate large-scale natural language corpora into training and extend the maximum sequence length.
Limitation.
Acknowledgment
This project is supported by the National Research Foundation, Singapore under its NRFF Award WNNRFF13-202100000 and the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2022).
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamerr.
NeurIPS, pages 1877-1901, 2020. 2,
[2] Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T Freeman, Maskgit: Masked generative image transformer. In CVPR, pages 1 11315 11325, 2022. 3,7
[3] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109 10052, 2021.
[4] Rodrigo de Bem, Arnab Ghosh, Thalaiyasingam Ajanthan, Ondrej Miksik, Adnane Boukhayma, N Siddharth, and Philip Torr Dgpose: Deep generative models for human body analysis. International Journal of Computer Vision, 123:1537, 2020.
[5] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: mobile app datasct for building data-driven design applications. In Proceedings of the 30th annual ACM symposiun on user interface software and technology, pages 845- 854, 2017. 5,6
[6] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint (rXXv::810004805: 2018. 2, 4
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scalc. In ICLR, 2021.
[8] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305. 15393, 2023. 3
[9] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pages 10696-10700 2022. 7 [10] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. pages 770-778, 2016. 1
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 30, 2017.
[12] Lianghua Huang. Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778. 2023, 1
[13] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra Mayu Otani, and Kota Yamaguchi. LayoutDM: Discrete Diffusion Model for Controllable Layout Generation. In CVPR, pages 10167-10176, 2023. 6,
[14] James M Joyce. Kullback-leibler divergence. In International encyclopedia of statistical science, pages
720-722. 2011.
[15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint
2tv:2304.006443. 2023. 1
[16] Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: bidirectional layout transformer for controllable layout generation. In ECCV, pages 474-490. Springer, 2022. 3,
[17] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 128:1956-1999
2020, 2,5, 2,5, 6
[18] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. Al- bert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 11942.
2019. 2,4 4
[19] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-scquence pro-training for natural language generation, translation, and comprehension. pages 7871-7880 2022.
[20] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language image pre-training with frozen image encoders and large language models. arXiv preprini arXiv:2301. 2tt.::22301111997,,. 2023.
[21] Jiefeng Li, Can Wang. Hao Zhu, Yihuan Mao. Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and new benchmark. In CVPR, pages 10863-10872, 2019. 2,5
[22] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded text-to-image generation. CVPR, 2023. 1, 4,9
[23] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco; Common objects in context. In ECCV, pages - 755, 2014. 2,
11

---

[24] Haozhe Liu, Bing Li, Haoqian Wu, Hanbang Liang, Yawen Huang, Yuexiang Li. Bernard Ghanem, and Yefeng Zheng. Combating mode collapse via offline manifold entropy estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8834-8842, 2023. 3
[25] Haozhe Liu, Wentian Zhang, Bing Li, Haoqian Wu, Nanjun He, Yawen Huang. Yuexiang Li, Bernard Ghanem, and Yefeng Zheng. Adaptivemix: Improving gan training via feature space shrinkage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16219-
16229, 2023. 3
[26] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv
preprint X907 1692. 2019. 2, 4
[27] Bryan A Plummer, Liwei Wang. Chris M Corvantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting models. In /CCV, pages 2641-2649 2015. 5, 6
sitoontt--phase correspondences for richer image-to-sentenes
[28] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding
by generative pre-training. 2018. 2, 4
[29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language
models unsupervised multitask learners. OpenAl blog, page 9, 2019. 2, 4
[30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022. 3
[31] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pages
2021. 3
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution
image synthesis with latent diffusion models. CVPR, pages 10684-10668 2022. 1,
[33] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large
Scale Visual Recognition Challenge. ISCV, 115:211-22 1-222, 2015. 6,6
[34] Chitwan Saharia, William Chan, Saurabh Saxcna, Lala Li, Jay Whang, Emily Denton, Scyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text- to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. / 1487, 2022.
[35] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.0844
2022. [36] Christoph Schubmann, Richard Veneu, Romain Bcaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-texi pairs. arXiv preprint 11.01:2111.021144 2021.
[37] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint 1508 07900, 2015. 4
[38] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, pages 8430-8439, 2019. 2,
5,
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977119277 2023.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, pages 5998-6008, 2017.
[41] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In CVPR, pages 12193-12202,
2020. 5
[42] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng. and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV, pages 7452- 14217 2023. 3
[43] Lvmin Zhang and Maneesh Agrawala, Adding conditional control to text-to-image diffuusion models. arXiv preprint arXi: 05544. 2023. 3,
[44] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout
analysis. In ICDAR, pages 1015-1022. IEEE, 2019. 5.
[45] Guangming Zhu, Liang Zhang. Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, et al. Scene graph generation: A comprehensive survey. arXiv preprint ar Xiv::2201.0000 00444, 2022. 3
[46] Mingchen Zhuge. Haozhe Liu, Francesco Faccio. Dylan R Ashley, Róbert Csoráás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Hermman. Kazuki Irie, et al. Mindstorms in natural language-basee societies of mind, arXiv preprint arXiv:2305.17066 2023. 2
12

---

6 Appendix
6.1 Advantages compared selecting GT bounding boxes from train set
(i) Diverse layouts generation. When users are required to specify a large number of classes (larger than 10), the layouts selected from COCO train set including all such classes limited. As shown in Fig. 3, when target classes ("bottle", "dining table', 'person', "knife', "bowl', "bowl', 'oven', 'person', 'cup'. "cup', 'bowl', 'bowl', 'broccoli', 'spoon') are considered, there is only one relevant layout containing all the target classes from COCO train set. In contrast, as shown in Fig. 3, VISORGPT can receive these target classes to infer lot of corresponding layouts with much diversity.
(ii) User-customized Controllability. Compared to selecting from ground truth bounding boxes, one of the main advantages of VISORGPT lies in its ability to extrapolate novel layouts based on users' configurations, as depicted in Fig. 10 (c). VISORGPT can deduce the missing classes' positions ('skis, skis, skis, skis") based on the layout ("person, person") given by users.
(iii) Free-form captions as layout condition. Incorporating captions in the training sequences enables VISORGPT to receive free-form natural language input as a condition (e.g. a boy and girl are playing by the sea or a train is running toward the man) and generate corresponding layouts (as shown in Fig. 10 (b)).
6.2 Extrapolation ability
To showcase VISORGPT's extrapolation ability, we present examples in Fig. 10 (a). As demonstrated, when the 'sea' occupies the majority of the canvas, the 'surfboard' can appear throughout the entire canvas. Upon shifting the 'sea' region to the middle part of the canvas, the 'surfboard' is accordingly confined within the middle region. This phenomenon demonstrates that VISORGPT does not solely rely on memorization; rather, learns intrinsic relation, location, and shape priors among categories. Consequently, VISORGPT can extrapolate unseen/novel layouts based on users' configurations.
6.3
Examples of Training Sequences
Here, we give some examples of various types of training sequences on different datasets:
Human Pose (COCO):
key point; multiple instances; large; I; 18; person; a 190 120 b 266 146 318 143 d 38385 232 e 338 269 f 214 150 g 0 0 312 280 365 296 359 4201 258 283 m 194 344 301 383 o 197 100 p 181 103 q 234 84 0
Human Pose (CrowdPose)::
key point; multiple instances; large: 2; 14; person, person; [ a 312 201 b 306 200 311 232 d 269 214
298 257 231 206 g 296 275 h 307 275 251 244 271 235 274 292 283 295 m 304 153 n 310 191] a
179 247 b 165 245 164 313 d 160 315 e 221 316 207 279 g 155 343 h 144 366 242 337 j 240 367 k
210 431 300 418 m 172 176 n 177 227] key point; multiple instances; large; 2; 14; person, person; [ a 240
178 b 304 168 e 228 239 0 0 261 236 f g 251 296 h 289 296 0 0 0 0 m 261 92 272
156] a 314 160 363 158 274 232 356 264 224 260 f 271 263 g 298 315 h 341 324 i 000 332 442 k
00100 0 m 287 64 n 333 133]
Instance Mask:
mask; multiple instances; medium; 1; 0; clock; [ m0 224 291 m1 226 299 m2 227 306 m3 228 313 m4 233 320 m5 238 325 m6 245 329 m7 252 332 m8 259 334 m9 266 335 m10 274 333 m11 281 330 m12 288 327
m13 293 323 m14 299 318 m15 303 312 m16 305 305 m17 307 298 m18 310 291 m19 308 284 m20 307
276 m21 303 269 m22 299 263 m23 295 257 m24 288 254 m25 280 251 m26 273 250 m27 266 249 m28
259 249 m29 252 251 m30 246 256 m31 240 260 m32 235 265 m33 229 270 m34 227 277 m35 225 284]
Object Centric Bounding Box: box; object centric; large; 1; 0; castle; xmin 236 ymin 142 xmax 413 ymax 232]
6.4 Implementation Details
All experimental evaluations were conducted on eight NVIDIA Tesla V100-32GB GPUs using PyTorch. In order to include special words, we created a new vocabulary containing a total of 30,769 words based on a standard vocabulary. To optimize computational efficiency and memory utilization, we utilized the DeepSpeed framework. To serialize visual locations, first resized the long side of each image to length of 522 pixels
13

---

and then shifted the image content to the center by padding the short side to length of 512 pixels. As result, the number of bins set 512. The flag of [Size] indicates the average area of all instances in the image and we set the flag according to the rule:
"small"
average 322 32 S average area <, 962 average area 2 962
"medium" "large"
We omitted person instances with fewer than five keypoints. To enable continuous generation, we designed and trained models based on the prompt format (b). Specifically, VISORGPT' (a&h) and VISORGPT (a&b) were trained using the same number of sequences as VISORGPT! (a) and VISORGPT (a), respectively. The only difference is that randomly utilized prompt format (a) (h) construct each training sequence.
During the evaluation stage, ensure efficient inference, In the ablation studies, we added special words only to the [Coordinate] term, and we reported the average KL divergence between the location and shape priors learned by VISORGPT and those in the real world. Since training large-scale language models is time- and resource- consuming. we trained only three types of VISORGPT with respect to GPT-2 (base, medium, large) with maximum token length of 256 in 50,000 iterations on COCO (Box) data.
the maximum sequence length of our model (VisORGPT) to 256 tokens
6.5 Evaluation Details
To estimate discrete visual prior from VISORGPT, we infer series of sequences via prompting as below:
Code in Python:
E"box; multiple instarces; random, choice small', 'medium', random.randint (2, 10); ; 0; category name, "
'large') ])
To ensure that each category in a given dataset sufficiently represented in the sequence data used for estimating the visual prior, we specify minimum number sequences which each category must appear. Table provides an overview of the predicted sequences that are used for evaluation.
Table 9: Details about the predicted sequences for evaluation.
Datasets
#Categories
#Predicted Seq.
Min #Seq. Per Category
Open Images (Box)
600
48,000
~80
Objects365 (Box)
365
29,200
~80
COCO (Box)
80
6,400
-80
In study, adopt the Kullback Leibler divergence quantify the similarity between two given discrete distributions. Specifically, let p and q denote estimated probabilistic priors derived from the real world data and the VISORGPT, respectively. The degree of similarity between these two distributions can be computed as:
(L(pllg) plog(p/q).
6.6 Visualization
Relation Prior of COCO. Fig. illustrates the comparison between the real and learned relation prior among 80 categories on the COCO dataset. As can be observed, there is a high degree of similarity between the two relation matrices.
More Visual Comparison. We provide more comparison of visual prior between the real world and one learned by our VISORGPT and failure cases on COCO dataset in Fig. 11.
Continuous Generation. Fig. 12 presents set examples showcasing continuous generation based on the current scene. Notably, in each row, the proposed VISORGPT is able to successfully complete a scene that involves many individuals annotated with 14/18 keypoints or objects with bounding boxes, based on the information provided in the corresponding scene depicted in the previous columns.
Figs. 13 and 14 present more visualization results.
14

---

Rea World
Figure 9: Relation among 80 categories on COCO.
(a) Prior of 'surfboard' adapts to variations of the 'sea' (b) Supports to receive natural language as input
(c) Scene completion based on user provided objects
(d) Multimodal inference of boxes and keypoints
Figure 10: More visual examples.
15

---

Airplane
Boat
Cat
Skateboard
Real World
VISORGPT
(a) Comparison of location prior
Apple
Backpack
Banana
Baseball glove
Bottle
WicktP
VesertP
Viser
WisitP
Bird
Boat
Bowl
Broccoli
Car
World
Seal Medil
Word
Hool moxk)
And
(b) Comparison of shape prior
Bear
Bicycle
Chair
Airplane
Book
Real Word
Real World
Wordd
Baseball bat
Cow
VISORGPT
Viseri
(b) Failure cases
Figure 11: Comparison of visual prior between the real world and one learned by VISORGPT on COCO dataset.
16

---

Continuous Generation based on the current scene
FAMM
Continuous generation COCO (keypoint)
Continuous generation on CrowdPose
Continuous generation COCO (Box)
Figure 12: Examples continual generation.
17

---

Iopat promen
beypoent mu mitarma ou langer permary, Sernurs perton, parmnn petsic, SEOSA pettin DO DETON
227
10 270
208 286 Lan 306 306 y 230 LGl e 207 16A 2 194 53 1 TON BA TO
TO 205 TH 255 BA 2A11 ES 3. CE 365 DE MOT
13
18:7
340 352 350 D 3 3224 201 122
250
220
CDD 230 LA3 L 377 100 750 LE0
220
459 385 862 (160 683 L 990 152 d54 136T BS0 (10 (2) 380 102 1861 B0T 317) 663 LEE LAE 31 5O0
457 1200 SPR 106 S06 1XD (O0 300 160 6 185
100 600-031
1RR (akig 178 197 257 10
240 d 010 300 100 10R 162 TOD I 176 1706
712
1ND LSA 178 143 124
246]
225
1F4 (66 196: (. 348 Litt 276 168 AOe 1 SNE 140 101 118 2350 100 220 13tN INC 1871 200 200) 2.0 250 351 U5A 200313 20B PN: 313
6.788 195 3771
136
O6 203 188 (-000 000
175
0-00 133-4
D 0.0 Ag0 y 4-436 327 0: 250 333 (OO) OU! 425 233 400 257 23 D OU0 4010-11332100 335 200 37S 5-300 350 327 LOS LLD 333 303 129 10 OST 329 52 2341 322 230 1115 203 8:0 199
Ovits
Biyy poliin res Rigal ixponces Buge R; 18 peenon, pertoe, parsoos penson, po pandan pensOe persic > FAAD O7 O 20 ER 211 COO BAO ON DJ 274
6.0 DVOj BK di ADE Net 42% M07 EN 3PV wnd 6 01 R 1oth 230 2000 208 370 4 0 R00 M UOS SOU FAN DON NoR BAR SNS AOG PAO M1P vn RES BoS 951 RE0 206 H4 202
D 1NG L BoR USI 301 32% 1.2 yra 160 pa 206 25. 286 2701 HET 241e M0 271 121 mon
tie ehi GA FOE 2TT 176 188 194 140 1 dic
11
422 40T 3TO 387 ZED P 230 215 3ED 234- 2ON BOD 240 DONOO 320 302 308 438 120 228 3O4 ZST 22A EST 337 200 O00)
Phpa prompt
kir poliin reua Rigil exzances; bugee R: 38 paeroon, pertoe, parmons penSOn, parcze, penoan penooe pernee
DULT & ON SAO
DO DO Bi D: 423 200 D02 325 w 0 B011 200 275 ZEN 224 6 001 300 2244
285
204 AOD 237 SED 301 AED 2014 150
334 C' 200 320 i 560 120 380 221 333 2 123 270 339 275 280 325 228 255 299 275 34T 313 D80 STE 404 COR SOD STT 4N3 SO0 LIN 115 374 245 DOD T0E 2022 1 DD
Pith 3
3R 27h 85p00
211 583
433 NOT 3TO
220 OO 330 225 202 224 CAN B00 0O0
220 202 308 200 438 220 3SA EOT 22A 337 200 OOO
Opea pranpe
bey puni me kigl motuancu renluam 3 36 (NocaA, pecton E0POIT, UNTSOR
2012 (2th 255
bex, A/2 d6th 12B di D.C 220 2/5
Mrm
L Diao 2.90 201 EVi 2001 abe 2.5 200//b 223 201 DM2 2.5 339 2401 283
en 257
347 1901 Za0
266
210-28---000 245 522
332
SXT 1/2 SUS en 333 1.50 333 35t U1D D1ggg 344
LX0 2.00 2,20 L C10 305
00.00:00 6.0 GC 3US 1.000000-0000 243 444 262-4
433 248 343
378 2031 301 31d UUS 300) 436
358 440 3011 330 300
304 3S1 306 287778777447 0.0
4100 340
1931443
PLANT ansegn
ber, poin: muliine instancaa arg. 3: perser person, zerton: a 104 220
103 235 353 Dng 334 LJ5 250 302 330: 104
0 210p
225 234
400 360 224
4&4 PiR CSE la 064 F33 390 N47 NOM NAT 9iS (077 9811 (-0 9177
1 943 204 414 300 11 246
912 301y
USS 299
ve
60n 307 300 240 & 934 (o
Figure 13: Examples of input prompts, output sequences, decoded results and synthetic images.
18

---

Input prompt:
multiple
table,
kniff, apple, apple, apple;
Outpot sequence:
instances; large; dining inifo, apple, apple, apple; | xmin ymin
512 421 [ xmin ymin 85 ymar ][ ymar 299 smin 366 ymin 254 xmax 472 203 241
ymin
ymax 274]
Inpot prompt:
multiple instances; 5; horse,
person, person, horse, person;
Outpot sequence:
large; persan, person, horse, person; xmin ymin 164 ymax
multiple
umin 302 198
ymin
xmax 315 ymax 327 ymax
xmin 305 ymin 190
203
xmax 353 ymax 297
Input prompt:
multpple instances; lange; 5;
horse, horse, harse, horse;
Output sequence:
instances; large; 0;
horse, horse, horse, horse; ximin 0 ymin 212
ymax
ymin
ymax 237 xmin ymin 175 xmax 180
225 xmin 235 144 512 ymax ymar 238
box multiple instances; large; 7: person, cup, donut, donut, donut, donut, dining table;
sequence:
box: multiple instances; large; 7: 7. person, cup,
donut, donut, donut, donut, dining table: ymin
ymax
37 ymin 136 xmax 183 ymax 116 jamin 235
144 352
339
ymin 243 xmax 479 ymax 385 xmin 212
ymin 239 xmax 357 ymax 378 xmin 354
ymin
ymax
ymin
xmax 511 ymax 442
imput prompt:
muttiple instances, large;
person, person, umbrolla, tie;
Outpot sequence:
box: multiple instances; large; 5; bost, person, person, umbrella, tie; xmin ymin ymax
ymin xmax 381 ymax 391 II smin 72 ymin 136 xmax xmin
243
98 ymin
Beebeh ViH.
ymax
ymax 250
Input prompt:
botc multiple instances; large; 5; vase, chair,
dining vase,
Outpot sequence:
multiple instances:
dining
chair,
ymin
xmax 305 ymax 411 I kmin 85 ymin 251 xmax
423
ymax s05 xmin 293 ymin 276 xmax 377
ymax 417 xmin 345 ymin 243 xmax 427
ymax
Figure 14: Examples of input prompts, output sequences, decoded results, and synthetic images.
19


Reviews:
[{"review_id": "NLmED8f1Hl", "reviewer": "Reviewer_KtmJ", "summary": "The paper proposes a method to learn a high-level generative model for images based on language modeling, which allows the generation of object masks, bounding boxes, and human pose keypoints. The paper proposes a novel tokenization scheme to turn annotations into token sequences that can be fed into an off-the-shelf language model. This tokenization scheme is targeted for these tasks and shows improved performance over alternative approaches. Quantitative experiments show that the model can capture the distribution of data on the datasets tested. Qualitative results show that the learned model is helpful for conditional image generation when compared against simple baselines for generating object bounding boxes.", "questions": "See weakness.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper leverages recent progress in language modeling to cast a generative vision problem as a language prediction problem, showing promising results that may be useful for conditional image generation.\n- The paper proposes a novel way of encoding high-level image generative prior, with unique tokens designed for the three tasks studied, and they show that it improves performance over alternatives.\n- Ablation studies test the main design choices. The tokenization strategy for the numbers is sound, and the paper shows that it improves over alternative strategies (for example, just tokenizing the text as done in LLM's without any special discretization for numbers or other special tokens). Ablations also show that finetuning from GPT-2 does not improve performance, with or without special tokens. Although a more powerful LLM model may overcome this, this is out of the scope of the paper, and GPT-2 seems a reasonable LLM for this ablation.\n- Qualitative results show that having access to a high-level generative image prior is helpful for conditional image generation.", "weaknesses": "- The multimodality of the model is not used in practice nor tested. Although the method can produce keypoints, masks and boxes, from the examples and quantitative analysis it does not seem that the model is tasked to produce a mix of them in either train or test time. With this, the model may be acting as three independent models, each one being active depending on the input prompt (mask/bbox/keypoint). What is the advantage of multimodal training and can the model be prompted to produce consistent masks/bboxes and keypoints at the same time? It seems possible to train such a model as COCO contains the three modalities studied, for example.\n- Related work includes a \"Conditional Image Synthesis\" section that is only tangentially relevant to the paper (for the qualitative applications in section 4.5). It does not include a section that analyses previous work on \"Generative high-level image priors\", which is the paper's main focus. There's plenty of work on generative models for images similar to the ones studied, which is disregarded, such as Scene graphs [1] and human keypoint generation [2], among others. \n- Lack of comparisons to previous works. Comparisons against previous works should be performed to see how the model compares against them, especially since the model does not seem to be multimodal in practice (see points above). I.e., comparing against methods such as [1,2]. \n- Some results highlighted are unsurprising and do not give valuable insights, considering the progress in language modeling the paper builds upon. It is expected that the model will be able to memorize the datasets and their distribution, as shown in Figures 3 and 4, so these figures seem redundant, as they show that the model can replicate the training dataset up to a certain fidelity. A more interesting figure/analysis would be whether the model can extrapolate to configurations not seen in training. For example, if the model is prompted to produce masks and tasked to complete a sequence where a \"sea\" region occupies the whole canvas, is the prior for \"surfboard\" all of the canvas? And if the sea occupies only the left side of the frame, does the surfboard occupy only that region?\n- Qualitative analysis of the trained model does not show its advantages compared to using the datasets it was trained on. How do Figure 5 and 6 show that the generative model is better than just using the datasets already available (for example, by selecting a random example that corresponds to the prompt)? An advantage would be if the model can be prompted with configurations that are not on the training set (but this is not specified in the section), or if the model can be used for completion (e.g., the prompt is five object bounding boxes, the user provides 3 instances and the model generates the other two and are consistent with the user provided ones), but these or similar settings are not studied.\n\n[1] Scene Graph Generation: A Comprehensive Survey. Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, Mohammed Bennamoun\n\n[2] DGPose: Deep Generative Models for Human Body Analysis. Rodrigo de Bem, Arnab Ghosh, Thalaiyasingam Ajanthan, Ondrej Miksik, Adnane Boukhayma, N. Siddharth & Philip Torr \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The Figures and Tables are provided in the global pdf.**  \n\n1. The multimodal training and inference.\n\n      Thanks for pointing out. We train VisorGPT using a mix of boxes and keypoints and present some examples in Fig. 1. One can observe that VisorGPT exhibits the capability to infer bounding boxes and keypoints simultaneously. For instance in Fig. 1(a), it can deduce the presence of two individuals and a frisbee along with their corresponding bounding boxes. Besides, the keypoints for these two identified people are also predicted at the same time. The observation confirms that VisorGPT is capable of engaging in multimodal inference. The integration of multimodal training confers an advantage to the model, enabling it to acquire a cohesive representation of boxes/masks and keypoints for various objects. We will include such experiments (a mix training of bbox/mask and keypoint) and release the corresponding models.  \n\n2. Related works.\n\n      Thank you for your suggestion. We acknowledge the significance of the related works [1,2] you mentioned and have decided to incorporate them as a dedicated subsection in our paper. Furthermore, we will provide comprehensive discussions on these related works to ensure a more thorough and contextual understanding of their relevance to our study.                                        \n\n3. Comparison to previous works. \n\n      Thank you for your suggestion. Scene graph generation takes an image as an input and generates a scene graph, however, VisorGPT receives only the text prompts to generate boxes/keypoints. Hence. they are different and hard to compare. Instead, to further enhance the evaluation, we reproduce recent SOTA methods for designing document and mobile application interfaces such as LayoutDM(CVPR’23)[3], and VQDiffusion(CVPR’22)[4] on COCO dataset for comparison with VisorGPT. Besides, we also apply VisorGPT to documents and mobile applications design benchmarks like Rico [5] and PubLayNet [6]. As shown in Table 1, the experimental results include FID and Align. score (referrenced from LayoutDM) with lower values indicates better performance. Notably, **VisorGPT significantly surpasses these SOTA methods with better FID and Align. score on all three datasets**, showcasing the superior capability of VisorGPT to learn layouts. We will include these results and more comparisons (including pose) in the paper. \n\n4. Extrapolation ability and concern about memorization of the dataset.\n\t\n      (i) To showcase VisorGPT's **extrapolation** ability, we present examples in Fig. 2. As demonstrated, when the ‘sea’ occupies the majority of the canvas, the ‘surfboard’ can appear throughout the entire canvas (Fig. 2(a)). Upon shifting the ‘sea’ region to the middle/left part of the canvas, the ‘surfboard’ is accordingly confined within the middle/left region (Figs. 2(b) and (c)). This phenomenon demonstrates that VisorGPT does not solely rely on memorization; rather, it learns intrinsic relation, location, and shape priors among categories. Consequently, **VisorGPT can extrapolate unseen/novel layouts based on users' configurations**.\n\n      (ii) To address concerns about **memorization**, we present examples in Fig.4. Firstly, we specify target classes (i.e., 'bottle', 'dining table', 'person', 'knife', 'bowl', 'oven', 'person', 'cup', 'cup', 'bowl', 'bowl', 'broccoli', 'spoon') and then select layouts satisfying these conditions from the training set (Fig. 4a, only one satisfied sample over 110K samples). In contrast, we use the same target classes as prompts for VisorGPT, inferring their bounding boxes. As depicted on the right in Fig. 4(a), the bounding boxes generated by VisorGPT exhibit diversity and considerable differences from that selected from the training set. This also validates that VisorGPT does not rely on memorizing the dataset; rather, it learns intrinsic priors among categories, facilitating the generation of novel and unseen layouts. We will include these results in the latest version of the paper.\n\n5. Advantages compared to direct selection of bounding boxes from training set and **scene completion**.\n   \n      i) When users need to specify a large number of categories (more than 10), only a few layouts in the dataset meet these requirements. For instance, in Figs. 4(a) and (b), there is only one layout in the training set containing all the required target classes. Moreover, the selected layouts from the dataset remain fixed. In contrast, as depicted on the right in Figs. 4(a) and (b), VisorGPT can receive the target classes and generate a lot of layouts with diversity that fulfill the requirements.\n\n      ii) Furthermore, our **VisorGPT facilitates interactive fulfillment of user demands**. For instance, users can provide simple layouts, and VisorGPT can subsequently augment them with additional layouts. Fig. 3 illustrates such examples. As shown in Fig. 3(a), the objective is to infer six instances ('person, person, skis, skis, skis, skis'), and users can initially provide two instances ('person, person'). VisorGPT then generates the remaining four instances (‘skis, skis, skis, skis’), with consistency to the user-provided ones. Notably, random selection from GT boxes fails to achieve such outcomes, as it cannot generate novel layouts with the diverse configurations demanded by users. Additional experimental results are presented in Appendix 1.4. We will include more examples and discussions on this matter.\n\n[1] Scene Graph Generation: A Comprehensive Survey.  \n[2] DGPose: Deep Generative Models for Human Body Analysis.   \n[3] Layoutdm: discrete diffusion model for controllable layout generation.  \n[4] Vector Quantized Diffusion Model for Text-to-Image Synthesis  \n[5] Rico: A Mobile App Dataset for Building Data-Driven Design Applications.  \n[6] PubLayNet: largest dataset ever for document layout analysis.\n"}, {"review_id": "sTCj3eh7HV", "reviewer": "Reviewer_6eEZ", "summary": "This is impressive work, striving to explicitly learn visual priors and enabling customization in the sampling process for conditional image synthesis. Such a task normally occurs inaccurate synthetic results due to improper conditional priors, such as spatial location, shape, and interaction of objects. \n\nTaking inspiration from the great success of generative language modeling that estimates the likelihood of a given sequence of words occurring in a sentence, the authors propose VisorGPT which converts visual priors into a corpus of sequences and casts the learning objective as maximizing the likelihood of each sequence. Two universal prompts are proposed to encode different types of visual locations, including object bounding box, human pose, and instance mask.\n\nOwing to the robust modeling capacity of generative pre-training, VisorGPT is successful in effectively modeling visual priors. This allows for the customization of sequential outputs through advanced prompt engineering.\n", "questions": "In addition to the above weaknesses, here are two more questions:\n\n- Despite VisorGPT being interesting and novel, why bother to use generative sequential pre-training to learn visual prior rather than other learning paradigms, such as GAN or VAE? Given that baseline works (may not exist) are not provided for comparison, what are the key advantages of VisorGPT?\n\n- Can we use generated images with learned prior as labeled data for training models for object detection, instance segmentation, and keypoint detection tasks?", "limitations": "The authors haven't adequately addressed the potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Scope and relevance**: The topic is very interesting and important. Considering the growing interest in AIGC, this paper fills the blank of learning visual priors for conditional image generation.\n\n- **Significance of contributions**:  The proposed model VisorGPT is novel and has expanded the territory of generative pre-training to learn visual priors, which mitigates the issue of lacking good prior in conditional image synthesis. \n\n- **Experiments**: The authors conduct extensive experiments and demonstrate improved generative results on conditional image generation.\n\n- **Clarity**:  The paper is well-written and easy to follow.", "weaknesses": "- **Closed-set bias**: As acknowledged by the authors, the overall categories are approximately 1000. Thus, the proposed VisorGPT may be biased toward closed-set training data. While I acknowledge that this is a pioneering attempt to learn visual priors with generative pre-training, I would recommend that the authors take care to provide additional guidance and precautions for the community in the paper.\n\n- **Data scalability**: The training data of VisorGPT is built upon existing labeled vision datasets. However, labeling these datasets with bounding boxes, instance masks, and key points is very time-consuming and expensive. Therefore, scaling up the training data is an important issue.\n\n- **Limited application**: The authors only focus the application of VisorGPT on conditional image generation. Are there any other potential applications that can be explored to show the usefulness of VisorGPT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The Figures and Tables are provided in the global pdf.**\n1. Add guidance and precaution to address the closed-set bias.  \n    Thanks for your valuable suggestion. Actually, 1,000 categories learned by VisorGPT can cover most of the common objects in the world. To include more potential categories, here we give some potential guidance. As we know, the large-scale text corpus contains much knowledge about the world and also the relation among different categories. Hence, if the text corpus is incorporated to train VisorGPT, the bounding boxes of unknown objects can be inferred based on the learned box priors and the relation among categories in the corpus. This is a potential direction toward open-world vocabulary situations. The second way is to scale up the dataset by automatically labeling using existing foundation models such as SAM(Segment Anything Model)[1]. We will include these potential discussions to the paper.  \n2. Data scalability.  \n    We are presently engaged in the expansion of the dataset by exploring SAM (Segment Anything Model)[1]. Leveraging the existing foundation models, we anticipate the creation of a more extensive and all-encompassing dataset to enhance the training of VisorGPT. Besides, leveraging text corpus in training may help to learn and infer more categories.\n\n3. Limited application.  \n    The primary objective of VisorGPT is to facilitate the generation of reasonable boxes, keypoints, and masks, serving as spatial conditions for guiding image synthesis. Additionally, there exist several sub-applications. For instance, in addition to the functions discussed in the paper, VisorGPT currently can interactively complete or refine coarse layouts provided by users, enhancing their reasonability—a crucial aspect for controllable image synthesis. This enables a more user-friendly image creation process. Furthermore, VisorGPT can be used to design printed media and application user interfaces (numerical results are shown in Table 1). We intend to incorporate these results in the latest version of the paper.  \n4. Why LLM rather than GAN or VAE? Show the advantages.  \n    VisorGPT offers three key advantages:  \n    i) Leveraging the transformer architecture, LLM demonstrates exceptional capabilities across various vision and language tasks.  \n    ii) Utilizing language modeling enables more flexible applications by prompting such as controlling the number/size/category to be generated and scene completion shown in Fig. 3.  \n    iii) Additionally, we conducted comparisons with recent state-of-the-art (SOTA) methods on designing document and mobile application interface layout namely LayoutDM (CVPR'23) [2], and VQDiffusion (CVPR'22) [3]. Notably, according to the literature, the performance of LayoutDM and VQDiffusion outperforms traditional GAN and VAE-based methods. As demonstrated in Table 1, **VisorGPT surpasses the SOTA methods on all three datasets**, including COCO, Rico[4], and PubLayNet[5]. The experimental results provide clear evidence of VisorGPT's superiority.  \n5. Synthetic data for downstream tasks.  \n    One of our objectives in this paper is to aid data synthesis for downstream tasks, such as object detection or pose estimation. Accordingly, we have designed the general prompt that enables users to address data imbalance issues in such downstream tasks, such as pose estimation in crowd scenes. For instance, by specifying the number of people larger than 10 in the prompt, the generated sequences can represent crowd scenes, facilitating the synthesis of images in such contexts. However, the current layout-to-image synthesis models exhibit limitations in synthesizing images that strictly adhering to the given layout. We believe that with advancements in these models, our VisorGPT can play a substantial role in automating data synthesis, thereby contributing significantly to the field.\n\n[1] Segment Anything.  \n[2] Layoutdm: discrete diffusion model for controllable layout generation.  \n[3] Vector Quantized Diffusion Model for Text-to-Image Synthesis.  \n[4] Rico: A Mobile App Dataset for Building Data-Driven Design Applications.  \n[5] PubLayNet: largest dataset ever for document layout analysis.  \n"}, {"review_id": "L6mwlTH1kL", "reviewer": "Reviewer_xtC6", "summary": "This paper proposes to translate image layout information such as bounding box layouts, masks and keypoints into discrete sequences that can be modelled by NLP architectures, such as GPT. By doing so, the image layout distribution of a dataset can be learned to later sample from it and obtain plausible in-distribution layouts. The experimental section shows that GPT learns to sequentially model these layouts successfully, and that the layouts can be later used as input to bounding box to image methods.", "questions": "Questions were asked above in the weaknesses section.\n\nOverall, one way of steering this work towards a useful application would be to use user text prompts, written as natural language sentences, and to show that they can be changed towards more in-distribution BB layouts to obtain better image quality, given the learned prior, with a consistent and quantitative evaluation. This would be a trade-off in controllability/image quality.", "limitations": "The paper addresses the limitations and societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method that this paper proposes to translate layouts into sequences is reasonable.\n- There has been efforts towards the reproducibility of the paper, although some details are missing. \n- The experimental section shows that the trained GPT model is successfully modelling the dataset distribution. \n- The output BB layouts and keypoints were used as input to generative models to obtain plausible generated images.", "weaknesses": "- The motivation and significance of this work is not clear and needs more work to ground it with a specific application.\n    - Why is learning the layout distribution of a dataset important? This idea works for in-distribution, but how is it useful when we want to generalize to unlikely or unseen layouts? With a creative user in mind, it is more than likely that unseen conditionings will be used. To enable controllability, generalization becomes really important. If the plan is to restrict conditionings to the in-distribution layouts, then controllability is sacrificed. Then, what is the end-goal of this work? \n    - Learning the distribution of the layouts is useful to sample conditionings that are very similar to the ground-truth (GT) BB layouts in the dataset. However, how is it different than filtering ground-truth BB layouts by specifying how many instances/sizes/category names we want? It would be interesting to see how the BB layouts and generated images compare when using the proposed method versus sampling GT layouts, across several samples given the same conditioning. Does the model provide more diversity and richer BB layouts than the GT ones? What is the real advantage of the model?\n    - The input prompts are not very user friendly and steer away from natural language. \n  \n\n- The evaluation section has significant room for improvement.\n    - It lacks comparison with any baseline. \n    - It lacks quantitative image quality/diversity/controllability comparison with other ways of obtaining bounding boxes and compared to text prompts. The most naive and simple baseline to obtain BB layouts is to use the GT ones.\n    - The paper presents several ways of generating image layouts (BB, masks, keypoints) but only BB layouts are evaluated.\n    - In Table 4, why using Ta + Tb instead of Tb alone? \n\n\n\n\n- The writing lacks clarity. Many details with respect to the evaluation are unclear or not properly conveyed in the text:\n    - l.59 \"...generalized visual intelligence models\". As mentioned above regarding the restriction of controllability, having this model output the BB layouts goes in the opposite direction of generalization. \n    - Section 3.1. I found it confusing to call the conditionings x, as they often referred to as y. The x variable is usually employed when talking about images. \n    - Figure 2. Given the examples of training sequences in Appendix 1.1, it is unclear how the instructions \"Add 2 person\" are learned. Lacking reproducibility details.\n    - Table 2. Unclear what \"sampling prop.\" and \"epochs\" mean here. Was the model trained on each dataset sequentially? Given the comments regarding the dataset balancing I do not believe it is the case, but this table induces confusion. \n    - Evaluation metrics. \"Format: whether it can be successfully decoded\" how is this measured?. \"Matching: number of categories matches the number of locations\" what does this mean?\n    - It seems \"shape prior\" is the same as \"aspect ratio\". Consider changing for clarity. \n    - Table 5. \"Quality\" and \"Controllability\" in this table could be confused by image quality and controllability, while this table is about matching statistics with the GT BB layouts. \n    - Table 7. Lack of context. What is SW, TK and #Seq in these experiments?\n    - How were the \"not obeying prior\" conditionings in Figure 5 obtained?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The Figures and Tables are provided in the global pdf**\n1. Generalize to novel layouts.  \nThe training set to train VisorGPT contains approximately 4 million images with around 40 million bounding boxes, providing **an extensive and diverse range of layouts to enable novel layout generation**. We utilize a set of target classes ('bottle', 'dining table', 'person', 'knife', 'bowl', 'bowl', 'oven', 'person', 'cup', 'cup', 'bowl', 'bowl', 'broccoli', 'spoon') to select the relevant layouts from the training set. We observe that only one satisfied sample over 110K samples contains all the target classes (as shown on the left in Fig. 4(a)).  \nSubsequently, we employ the same set of target classes as input for VisorGPT and infer their corresponding bounding boxes (as depicted on the right in Fig. 4(a)). Evidently, **the layouts generated by VisorGPT are diverse and differ significantly from the relevant layout selected from the training set**. This validates that VisorGPT does not rely on memorization. Instead, it learns intrinsic visual priors among categories, empowering it to infer novel/unseen layouts.  \nMoreover, **VisorGPT can receive user conditions, and reasonably complete the corresponding missing classes in an interactive way**. As depicted in Fig. 3(a), users can roughly draw two instances of 'person' across the canvas and prompt VisorGPT to deduce the remaining 'skis'. As demonstrated in Fig. 3(b), VisorGPT successfully infers the correct positions of four 'skis'. This validates that our **VisorGPT possesses robust generalization capabilities to accommodate the diverse requirements of users**.  \n2. Avantages and diversity comparison to selecting GT bounding boxes from training set.  \n(i) **Diverse layouts generation**. When users require to specify a large number of classes (larger than 10), the layouts selected from training set including all such classes are limited. As shown on the left in Fig. 4(a), when target classes ('bottle', 'dining table', 'person', 'knife', 'bowl', 'bowl', 'oven', 'person', 'cup', 'cup', 'bowl', 'bowl', 'broccoli', 'spoon') are considered, there is only one relavant layout containing all the target classes from the training set. Another example is shown in Fig. 4(b). In contrast, as shown on the right of Fig. 4, VisorGPT can receive these target classes to infer a lot of corresponding layouts with much diversity.  \n(ii) **User-customized/Controllability**. Compared to selecting from ground truth bounding boxes, one of the main advantages of VisorGPT lies in its ability to extrapolate novel layouts based on users' configurations, as depicted in Fig. 3. VisorGPT can deduce the missing classes’ positions (‘skis, skis, skis, skis’) based on the layout (‘person, person’) given by users.  \n(iii) **Free-form captions as layout condition**. Incorporating captions in the training sequences enables VisorGPT to receive free-form natural language input as condition (e.g.,A boy and a girl are playing by the sea) and generate corresponding layouts (as shown in Fig. 5). We will include these results in the paper.  \n3. Natural language input.  \nWe have extended VisorGPT to accommodate natural language input. As depicted in Fig. 5, when provided with the prompt \"a girl and a boy are playing by the sea\" VisorGPT can accurately infer the individual and background along with their respective bounding boxes. We will incorporate these results in the paper.  \n4. Comparison to baseline.  \nThanks for your suggestion. To enhance the evaluation, we reproduce recent SOTA methods such as LayoutDM(CVPR’23)[1] for designing layout of documents and mobile applications and VQDiffusion(CVPR’22)[2] on the COCO dataset for comparison. We also apply VisorGPT to documents and mobile applications designing benchmarks like Rico[3] and PubLayNet[4].The results are presented in Table 1(global pdf), which include FID and Align. score (referred from LayoutDM) with lower values indicates better performance. Notably, **VisorGPT significantly surpasses these methods on all three datasets**. We will include these results and more evaluations (mask and pose) in the paper.   \n5. Image quality/controllability comparison to other ways of box generation like the selection of gt ones.   \nAs suggested, we compare the baseline, which involves using GT boxes to synthesize images, in Table 2. Specifically, we utilize the bounding boxes from the COCO val set as spatial conditions for GLIGEN to synthesize images and use the bounding boxes generated by VisorGPT for GLIGEN to synthesize images. Then, we calculate both the FID and YOLO score [5]. As shown in Table 2, the FID and YOLO score of images produced using the bounding boxes generated by VisorGPT are similar to those obtained using GT boxes. This demonstrates that our generated bounding boxes closely adhere to the real distribution.  \nTable 2. Image comparison using GT Bboxes.\n| Method|FID $\\downarrow$ |YOLO score (AP)$\\uparrow$|  \n|:-:|:-:|:-:|   \n|GT Bbox |25.62|27.4|  \n|VisorGPT|26.37| 26.2|  \n\n6. Why use Ta + Tb instead of Tb alone?  \nFor users, Ta is simpler and more versatile than Tb, and Tb is specifically designed for scene completion (as shown in Fig. 3). For clear comparison, we will include results of using Tb alone.\n\n7. Clarity  \nDue to the limited maximum length, we cannot answer all the questions. We construct a large dataset to train VisorGPT, and the training is not sequential. We sincerely thank the reviewer for pointing out this lack of clarity, and we will add more descriptions and use clearer language to enhance the clarity of the paper.\n\n[1] Layoutdm: discrete diffusion model for controllable layout generation  \n[2] Vector Quantized Diffusion Model for Text-to-Image Synthesis  \n[3] Rico: A Mobile App Dataset for Building Data-Driven Design Applications  \n[4] PubLayNet: largest dataset ever for document layout analysis  \n[5] Image synthesis from layout with localityaware mask adaption  \n"}, {"review_id": "mClBPypOrw", "reviewer": "Reviewer_CQCW", "summary": "The paper proposes VisorGPT to model the propabilistic visual prior via generative pretraining. This is achieved by transforming the continuous values (locations, bounding boxes) into discrete tokens. Prompt engineering are employed to arrange these tokens into a sequence. The author conducts experiments to prove that VisorGPT is able to model the visual prior effectively and boost various downstream tasks like conditional image generation.", "questions": "(1) As stated in the weakness session, I am curious about the motivation to model the visual prior via GPT.\n\n(2) In experiment setting, the authors use KL divergence as a metric. However, I cannot figure out how to calculate the KL score, like how to estimate the distribution over the real bounding boxes? \n\n", "limitations": "The author has discussed about the limitations of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper proposes a novel setting: modeling visual prior via generative pretraining. This could be a potential area to combine visual prior with the power of large language models.\n\n(2) The paper is easy to follow, and includes comprehensive quantitative experiments and good visualization.", "weaknesses": "(1) The rationale for utilizing GPT to model visual priors is not well-explained. Given that many visual priors, such as bounding boxes and point locations, are continuous in nature, a straightforward baseline could be an autoregressive model with a regression head. It would be beneficial to provide comparisons with baseline methods and justification for using GPT in visual prior modeling.\n\n(2) The controllability of the proposed VisorGPT method appears to be overstated. While it can manipulate the number, size, and type of objects, the user is unable to control the position of each generated object or their relative positions. Furthermore, existing research (e.g., LayoutGPT [1]) has attempted to control the number, size, and type of objects using language models without additional training. As a result, the impact of VisorGPT may be somewhat limited.\n\n[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The Figures and Tables are provided in the global pdf**\n1. The rationale and motivation for utilizing GPT to model visual priors.  \nThere are three main considerations:  \ni) **Visual priors are not just location**. Our visual priors scope extends beyond the modeling of pure continuous numbers (e.g., location). Instead, we aim to unify locations, categories, and texts into a sequence representation, enabling a straightforward learning process without the need to design additional modules to adapt/process different types of modalities of information.  \nii) **Textual semantic understanding**. With the capacity to model locations, categories, and texts together, the model can learn synonyms of categories and relationships among different categories from the texts, enabling the extrapolation of synonyms or new categories based on the acquired priors of bounding boxes. Hence, the model is capable of addressing open-vocabulary scenarios. We present examples in Fig. 5, where 'girl,' 'boy,' and 'man' serve as synonyms for 'person.' These results will be included in the paper.  \niii) **Fulfill requirements by prompting and completing**. The Generative Pre-Training (GPT) paradigm facilitates flexible and personalized applications through prompting. For instance, by inputting a sequence like 'box;' or 'keypoint;' to VisorGPT, it can generate corresponding box or keypoint information. Additionally, as depicted in Fig. 3(a), we can provide initial layouts ('person, person') for VisorGPT, which then infers the missing classes along with their bounding boxes (Fig. 3(b)). These functionalities are attributed to the learning paradigm, i.e., modeling all modalities within a sequence.  \n2. Comparisons to baseline methods.  \nTo enhance the evaluation, we reproduce recent SOTA methods such as LayoutDM(CVPR’23) [1] for designing the layout of documents and mobile applications, and VQDiffusion(CVPR’22) [2] on COCO dataset for comparison. We also apply VisorGPT to documents and mobile applications designing benchmarks like Rico [3] and PubLayNet [4]. As shown in Table 1, the experimental results include FID and Align. score (referred from LayoutDM) with lower values indicates better performance. Notably, **VisorGPT significantly surpasses these baseline and SOTA methods on all three datasets**, showcasing the superior capability of VisorGPT to learn layout generation. We will include these results in the paper.  \n3. The controllability of VisorGPT.   \nApart from controlling the number, size, and type of objects, VisorGPT can also control the position of objects in three aspects:  \ni) As shown in Fig. 3, **users can control some objects’ positions by first providing their bounding boxes**, and the VisorGPT can receive these information to infer a complete layouts. In Fig. 3(a), when the objective is to infer six instances ('person, person, skis, skis, skis, skis'), users can provide positions of two instances ('person, person') and VisorGPT can then generates the remaining four instances (Fig. 3(b)), with consistency to the user-provided ones.  \nii) Besides, **VisorGPT currently can receive natural language input**. Hence, users can control the relative positions among objects by using natural language. As shown in Fig. 5, given input text prompt “A boy and a girl are playing by the sea.”, VisorGPT can output corresponding bounding boxes adhering to such relative positions.  \niii) Further, we have extended VisorGPT to receive users’ provided coarse layouts as input and refine such coarse layouts to a more reasonable ones based on the learned priors. Hence, users can potentially control each object’s position.   \nThese results will be included in the latest version of paper.   \n4. Comparison to the concurrent work LayoutGPT.   \ni) LayoutGPT reasons layout by textual knowledge and in-context examples. When encountering very complex scenes involving a large number of objects or segmentation masks, LLM textual knowledge is challenging to understand visual layout from limited examples with dense numerical data. In contrast, VisorGPT, with large-scale pretraining on natural layout corpus, can generate not only numerous objects (as depicted on the right in Fig. 4) but also segmentation masks.  \nii) Compared to LayoutGPT which generates the layout by starting from fix-style input), training-based VisorGPT can complete the layout based on in-consistent conditions (e.g., starting by category, numerical box, or key points), which offers greater personalization potential for condition generation. For instance, as shown in Fig. 3, the Tb prompt format (introduced in section 3.2) empowers VisorGPT to perform scene completion based on users' provided pre-defined layouts.  \niii) **LayoutGPT relies on the paid GPT-3.5/4 API, incurring costs for users**. On the other hand, based on publicly available GPT-2, VisorGPT will be freely accessible.  \nWe will incorporate these relevant discussions about LayoutGPT in the paper.\n5. How to calculate the KL DIV metric?   \nFor example, to estimate the shape prior of a category such as 'dog' we calculate the ratio of width to height of all bounding boxes associated with dogs in the test set, generating a list of numerical results that we use to construct a histogram representing the estimated distribution. Similarly, for VisorGPT, we can employ the same method to estimate a distribution, and subsequently calculate the KL divergence between real-world  and VisorGPT distributions.  \nAdditionally, we enhance our evaluation by incorporating FID and Align scores from relevant literature on document and mobile application design (Table 1). The results will be included in the paper.  \n\n[1] Layoutdm: discrete diffusion model for controllable layout generation  \n[2] Vector Quantized Diffusion Model for Text-to-Image Synthesis  \n[3] Rico: A Mobile App Dataset for Building Data-Driven Design Applications  \n[4] PubLayNet: largest dataset ever for document layout analysis  "}]

Author rebuttal:
**The Figures and Tables are provided in the global pdf.**

Meta-review: The paper is interesting, and its focus is timely and important, given the continuing rapid rise of visual synthesis (and their dependence of prior inputs).  Reviews raised some questions and most of them has been resolved during the rebuttal. Finally, all reviewers recommend acceptance, to varying degree. The paper will be a valuable contribution to the program.
Decision: Accept (poster)