SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinheng Xie $^{1}$  Kai Ye $^{2*}$  Yudong Li $^{2*}$  Yuexiang Li $^{3}$  Kevin Qinghong Lin $^{1}$  Yefeng Zheng $^{3}$  Linlin Shen $^{2}$  Mike Zheng Shou $^{1\dagger}$ $^{1}$  Show Lab, National University of Singapore  $^{2}$  Shenzhen University  
 $^{3}$  Jarvis Research Center, Tencent YouTu Lab  
{sierkinhane,mike.zheng.shou}@gmail.com  
https://sierkinhane.github.io/visor-gpt
Various stuff and things in visual data possess specific traits, which can be learned by deep neural networks and are implicitly represented as the visual prior, e.g., object location and shape, in the model. Such prior potentially impacts many vision tasks. For example, in conditional image synthesis, spatial conditions failing to adhere to the prior can result in visually inaccurate synthetic results. This work aims to explicitly learn the visual prior and enable the customization of sampling. Inspired by advances in language modeling, we propose to learn Visual prior via Generative Pre-Training, dubbed VISORGPT. By discretizing visual locations, e.g., bounding boxes, human pose, and instance masks, into sequences, VISORGPT can model visual prior through likelihood maximization. Besides, prompt engineering is investigated to unify various visual locations and enable customized sampling of sequential outputs from the learned prior. Experimental results demonstrate the effectiveness of VISORGPT in modeling visual prior and extrapolating to novel scenes, potentially motivating that discrete visual locations can be integrated into the learning paradigm of current language models to further perceive visual world.
The digital camera can continuously capture photographs of the visual world, such that tremendous photos and videos are currently shared on the Internet. In our world, various stuff and things possess specific traits, which have been correspondingly embedded in such visual data. In the current era of deep learning, deep neural networks [10, 40, 7] have demonstrated remarkable proficiency in learning from vast amounts of data, leading to the development of visual foundation models (VFMs) [15, 32, 43, 12, 22, 20]. Such traits have been accordingly learned and implicitly represented as the visual prior in VFMs, which has the potential to impact real-world applications. An example that highlights its importance can be seen in the field of image synthesis. To present high-quality and natural-looking images, the synthetic stuff and things must adhere to the visual prior such as the spatial location, shape, and interaction of objects (Fig. 1 (a)). A vivid example of layout-to-image is provided in Fig. 1 (b). When the spatial conditions do not adhere to the visual prior, such as the shape of 'donut' not being square, the size of 'person' being similar to that of 'donut', and 'donut' being floated in the air instead of being placed on 'dining table', the resulting synthetic contents may be inaccurate and visually inconsistent with the desired outcome. Despite recent advances in conditional image synthesis such as ControlNet [43] and GLIGEN [22], the challenge of continuously sampling customized spatial conditions that adhere to the visual prior remains a difficult problem, particularly for automatic synthesis of massive images with corresponding fine-grained annotations.
In this paper, we study the problem of how to explicitly learn visual prior from the real world and enable customization of sampling. If we would like to paint a series of instances on a canvas, we
* Equal Contribution † Corresponding Author
(d) Learning Visual Prior via Generative Pre-Training
(e) Customized sampling from VISORGPT
Figure 1: An overview of the problem of visual prior (top) and VISORGPT (bottom). (a) refers to visual prior, e.g., location, shape, and relations of objects. (b) provides a failure case of image synthesis from spatial conditions that do not adhere to the prior. Specifically, the shape of the 'donut' not being square and 'donut' being floated in the air instead of being placed on 'dining table'. (c) displays a success case that conditions sampled from VISORGPT leads to a more accurate synthetic results. (d) illustrates that VISORGPT learns visual prior through sequence corpus converted from the visual world. (e) gives an example that a user customizes a sampling from VISORGPT by prompting. should decide what to paint and also their shapes, locations, interactions, etc. It seems that these elements share a joint probabilistic prior, in which any stuff or things can be accordingly sampled to construct a scene. As there may be many potential variables in the prior, it is extremely hard to be comprehensively formulated. Over the past few years, significant advances have been made in language modeling [28, 29, 1, 6, 46], demonstrating their remarkable capacity for modeling the probabilistic distribution of sentences. Our focus is on learning the visual prior of location, shape, and relationships among categories, rather than raw pixels. It is possible to convert such visual information into a series of sequences, such that the visual prior can be learned by language modeling. To this end, as presented in Fig. 1 (d), we propose to learn Visual prior via Generative Pre-Training, dubbed VISORGPT. Thanks to the development of deep learning, many high-quality annotated data such as bounding-box [23, 38, 17], human pose [23, 21], instance mask [23] are publicly available. This provides sufficient location, shape, and relation information of stuff and things in the visual world. Since they are all encoded using 2D or 3D coordinates, we can simply convert them into a corpus of sequences. In this way, the visual prior can be learned by a pretext objective, e.g., maximizing the likelihood of each sequence. Beyond this, prompt engineering is investigated to unify various visual locations and enable the customized sampling of sequential outputs from the learned prior.
As shown in Fig. 1 (e), according to the user's prompt, VISORGPT can correspondingly sample a sequence from the learned prior, which can be spatially decoded for image synthesis (Fig. 1 (c)). Since the decoded conditions adhere to the prior, the synthetic 'cup', 'dining table', and 'donut' are realistic and consistent with the desired semantics. This finding confirms that we can continuously customize spatial conditions from many aspects, e.g., data type, object size, number of instances, and classes, using VISORGPT. With the advance of conditional image synthesis, it is feasible to generate an endless supply of synthetic images with their corresponding fine-grained annotations, potentially providing ample resources to train more robust and generalized visual intelligence models.
Language Modeling. Language modeling aims to estimate the probability of a given sequence of words occurring in a sentence. In recent years, Large language models (LLMs) such as GPT series [28, 29, 1] and BERT family [6, 18, 26] have revolutionized the field of natural language processing. In particular, BERT family adopts the encoder(-decoder) architecture and employs masked language modeling techniques to model each given sentence bi-directionally in context. In contrast, GPT series employs the decoder-only architecture to sequentially model the probability of
next tokens by maximizing the likelihood of given sentences. Such a pretext objective allows for easy scaling up in terms of the model's parameters and training corpus. In this work, inspired by GPT series, we investigate the potential of a decoder-only architecture in modeling visual priors.
Layout Generation. Several related works [45, 4, 16, 2, 13] have focused on scene graphs or layout generation, specifically in the context of mobile interfaces and documents. For instance, Inoue et al. [13] addressed structural data using discrete diffusion models and provided a unified approach to diverse layout generation tasks. More recently, Feng et al. [8] directly delved into layout generation using LLMs via in-context learning. In contrast to these methods, our approach goes beyond the layout of interfaces and documents. We extend our focus to encompass various modalities, including human keypoints and semantic masks, and unify the modeling of this diverse information within a single straightforward learning objective.
Conditional Image Synthesis. With large-scale image-text datasets [36, 35], generative models [31, 30, 34, 32, 25, 24], e.g., DALL-E,Imagen, and Stable Diffusion, have shown a significant capacity for synthesizing images of higher quality and greater diversity. Recently, more controllable image synthesis models such as training-based ControlNet [43] and GLIGEN [22] and training-free BoxDiff [42], have demonstrated a remarkable ability to control the synthetic contents precisely. When it comes to generating an extensive set of novel images, relying solely on spatial conditions from users or referring from images is inefficient. To tackle this problem, our VISORGPT is capable of continuously sampling customized and novel spatial conditions, making it possible to synthesize endless streams of data for various practical applications.
In this section, we begin by presenting our problem formulation (§ 3.1) and prompt designs (§ 3.2) for unifying various visual information (e.g., class and location) as textual sequences. Building upon this, we introduce our model architecture and pretext objective (§ 3.3) to model the visual prior. Finally, we provide practical examples of how to sample customized sequential outputs from VISORGPT (§ 3.4).
We assume that visual location  $\pmb{x}$ , e.g., object bounding-box, human pose, and instance mask, follow a probabilistic prior distribution  $p_{\pmb{x}}$ . However, since  $p_{\pmb{x}}$  is often unavailable in practice, this work aims to learn a model  $f_{\Theta}$  with parameters  $\Theta$  that can empirically approximate the latent probabilistic prior  $p_{\pmb{x}}$ . By doing so, we can sample new instances  $\tilde{\pmb{x}}$  from the learned prior, denoted as  $\tilde{\pmb{x}} \sim f_{\Theta}$ , to facilitate various vision tasks, such as conditional image synthesis and action generation.
3.2 Visual Location as Sequences
In the discrete language domain, we witness that a variety of tasks, e.g., translation and question-answering, can be integrated as a unified template (e.g., prompts and responses) and then processed by one language model using different user prompts. However, when it comes to annotations of visual locations such as 2D object bounding-box, instance mask, and 3D human pose which are continuous and highly structured, a unified approach has yet to be explored and our objective is to investigate potential solutions to this issue. Following Chen et al. [3], we discretize visual annotations, i.e., continuous numbers, into  $m$  bins, such that location information can also be naturally represented as discrete tokens and  $m$  integers are then accordingly added to the standard vocabulary. It means that coordinates can be represented by a sequence of words. In particular, each number representing visual localization will be quantified as an integer in the range of  $[1, m]$ . In this way, visual locations  $x$  of each image can be then unified into a sequence  $\mathbf{t} = \mathrm{PROMPT}(x)$ .
As visual annotations of various tasks are in different formats, we propose three universal prompts:
Prompt template  $\mathrm{T}_{a}$ :
Annotation type; Data type; Size; #Instances; #Keypoints; Category names; Coordinates
Prompt template  $\mathrm{T}_b$
Annotation type; Data type; Size; #Instances; #Keypoints; [Category name i Coordinate i]
Prompt template  $\mathrm{T}_{\mathrm{c}}$  ..
Annotation type; Natural Language Input; Category names; Coordinates; Instruction; Category names; Coordinates
The provided prompts can be summarized in Tab. 1, which provides standardized templates to unify commonly used 2D and 3D visual location information into 1D textual sequences. Each prompt begins with the flags [Annotation type] and [Data type], which are the flags indicating the type of annotation and scene, e.g., box and multiple instances. The follow-
Table 1: Candidate choices of prompt template.
Annotation type	box; keypoint; mask; multimodal;···
Data type	object centric; multiple instances
Size	small; medium; large
#Instances	1; 2; 3; ···
#Keypoints	14; 18
Category name	cup; person; dog; ···
ing flags of [Size] and [#Instances] represent the average area and the number of instances in the current image, while [#Keypoints] indicates the number of keypoints annotated for each person, i.e., 14 or 18. The following two flags are the [Category name] of each instance and their corresponding [Coordinate]. At last, [Natural Language Input] is extended for free-form language input and [Instruction] is used for iterative refinement of the current layout. Examples of these prompts are presented in the following sections. By employing our defined templates, we transform commonly used visual annotations into a large-scale sequential corpus. The corpus can be seamlessly ingested by language models, facilitating better learning of visual commonsense prior.
3.3 Learning Visual Prior via Generative Pre-Training
Model Architecture. In the past few years, many large language models have been successively proposed, such as GPT [28, 29, 1] and BERT [6, 18, 26] family, and recently introduced LLaMA [39]. We employ the GPT decoder-style transformer as our model to learn the visual probabilistic prior.
**Predicate Objectivity.** After processing the visual locations  $x$  as textual sequences  $\mathbf{t}$  in § 3.2, we stack each sequence by byte-pair encoding (BPE) algorithm [37] to obtain a sequence with  $n$  tokens  $\mathbf{u} = \{u_1, u_2, \dots, u_n\}$  such that a standard language modeling objective can be directly employed to learn visual prior by maximizing the following likelihood:
$$
\mathcal {L} = \sum_ {i} \log p \left(u _ {i} \mid u _ {i - k}, \dots , u _ {i - 1}; \Theta\right), \tag {1}
$$
where  $k$  is the size of context window, and  $p(\cdot|\cdot)$  indicates the conditional probability which is modeled by the neural network  $\Theta$ . Stochastic gradient descent is used to train the neural network.
3.4 Customizing Sequential Output
In addition to offering formatted visual annotations for learning a probabilistic prior, the standardized templates enable to personalize sequential output for various applications through prompting. For example, the customized sequential output can be employed as spatial conditions in image synthesis models (e.g., ControlNet [43] and GLIGEN [22]). This opens up the possibility of synthesizing a broad range of data types to address diverse problems and challenges in computer vision. Here are a few representative scenarios:
(a) Object Bounding-Box. As we use a flag to distinguish different types of visual annotations, we can control the type of data and scene to be sampled from the learned probabilistic prior by setting the beginning tokens in the input prompt. Accordingly, we can set the beginning prompt as "box;" to generate sequential output with instances and corresponding bounding-box information. Besides, with flags like [Size], [#Instances], and [#Keypoints], we can sample a scene that adheres to multiple conditions. As depicted in Fig. 2 (a), we can input a prompt "box; multiple instances; small; 16; 0; kite, kite, person," as a prefix to require the VISORGPT to conditionally infer the remaining tokens. In this example, VISORGPT outputs the categories and their locations, specifically fulfilling the requirement of objects being in small size. In particular, (xmin, ymin) and (xmax, ymax) are special tokens indicating the top-left and bottom-right corners of the target object.
(b) Human Pose. With flags of [#Instances] and [#Keypoints], VISORGPT is capable of customizing sequential outputs involving instances with keypoints in a crowd scene. We give an example in Fig. 2 (b). Numbers (10 and 14) are added to the beginning of prompt as conditions to infer a scene consisting of 10 people with 14 keypoints. Note that, we use “a, b, c, d, …” and “m0, m1, m2, m3, …” as special tokens to distinguish each human keypoint and object boundary coordinate, respectively.
(c) Instance Mask. Beyond sparse coordinates as shown in (a) and (b), VISORGPT can deal with dense spatial annotations, i.e., instance masks. Typically, pixel-level information can be represented
Figure 2: Examples of customizing sequential outputs from the proposed VISORGPT.
using a mask matrix or a set of boundary coordinates. For convenient sequentialization, we uniformly sample  $n$  points along the angle in the polar space from object boundary coordinates to represent the pixel-level location, which is similar to [41]. We provide an example in Fig. 2 (c).
(d) Multimodal Inference. VISORGPT exhibits the capability of multimodal inference. In Fig. 2 (d), VISORGPT can deduce the presence of two individuals and a frisbee along with their bounding boxes and keypoints of these two people are also predicted at the same time.
(e) Natural Language Input and Instruction. Apart from the above prompts, VISORGPT can receive free-form language as input to generate corresponding layouts. We present an example in Fig. 2 (d). In addition, the proposed VISORGPT supports iterative refinement by instructions like "make the boy and girl closer".
Datasets. We collect around 4 million sequences from the publicly available datasets for VISORGPT. In particular, we consider three types of commonly used visual annotations, i.e., object bounding-box, human pose, and instance mask. In the MS-COCO dataset [23], we collect
Table 2: Training corpus for VISORGPT.
Datasets (type)	#Categories	#Images
Open Images (Box) [17]	600	1,743,042
Objects365 (Box) [38]	365	1,728,775
COCO (Box) [23]	80	117,266
ImageNet (Box) [33]	1,000	38,285
COCO (Keypoint) [23]	1	53,473
CrowdPose (Keypoint) [21]	1	9,981
COCO (Mask) [23]	80	117,266
Flickr30K [27]	44,518	31,783
Rico [5]	25	35,851
PubMedLayNet [44]	5	315,757
$\sim 118\mathrm{K}$  images annotated with 80 categories and their object bounding-boxes and instance masks. For each image, all object bounding-boxes and instance masks with their category information are formatted to a sequence, respectively. Beyond that,  $\sim 3.5$  million bounding-box annotations of Objects365 [38] and Open Images [17] are also converted to sequences. Other types of annotations (i.e., human keypoint) of MS-COCO ( $\sim 54\mathrm{K}$ ) and CrowdPose ( $\sim 10\mathrm{K}$ ) are also formatted to sequential data. For the object-centric scenario, we collect  $\sim 4\mathrm{K}$  sequences from ImageNet-1K [33]. Besides, Flickr30K [27] is employed to support free-form natural language as input. For the scenarios of mobile interface and document layout, we use the Rico [5] and PubLayNet [44] datasets. A summary is presented in Tab. 2.
Evaluation Metrics. We propose to evaluate VISORGPT from three aspects: (i) Evaluating the quality of sequences generated by VISORGPT. In the inference stage, as VISORGPT predicts sequences in the format given in § 3.2, it is necessary to examine whether the generated sequences can be decoded into visual locations. In particular, we generate a series of sequences using VISORGPT and calculate the accuracy whether it can be successfully decoded (termed Format in Table 5) and the number of categories matches the number of locations (termed Matching in Table 5). (ii) As discussed in § 3.2, we use flags, i.e., [Size] and [#Instances], to indicate the average size and number of instances in the current sequence. Hence, we can control the average object size and the number of instances in the generated sequences via setting flags [Size] and [#Instances]. Then, we can calculate the accuracy whether the object size and the number of instances in the generated sequences are consistent with the given flags to validate the performance of controllability (termed Size and #Instances, respectively). (iii) Evaluating the learned probabilistic prior, i.e., object location, shape, and relation among categories, on the val set of COCO, Objects365, and Open Images datasets. In this work, we propose to compare the discrete distribution of every visual prior. Specifically, to compute the location prior of a category, we initialize an empty canvas and convert the bounding-box of each instance of the category to a binary mask. Then, each mask is accumulated on the canvas and normalized as 2D location distribution. To compute the shape prior of a category, we calculate the ratio of width to height of each instance of the category, and estimate a discrete distribution as the shape prior. To establish the relation prior of a category to other categories, we count the number of co-occurrences between the category and other categories and estimate a discrete distribution. In this way, discrete prior of each category can be computed on COCO, Objects365, and OpenImages val sets as real one. Durring evaluation, we infer a series of sequences to compute the learned visual prior. Then we measure the similarity between learned and the real prior using the Kullback-Leibler divergence [14]. In addition, FID [11] is adopted to compare with layout generation methods [16, 13].
Table 3: Model card of VISORGPT.
Models	#Parameters	#Training data	Annotation type	Batch size	Iterations	Learning rate	Sequence length n
VISORGPT	117M	4M	box & keypoint & mask	128	200K	5.0e-5	1024
VISORGPT†	117M	34K	box & keypoint & mask	128	200K	5.0e-5	1024
Implementation Details. We provide training details of VISORGPT in Tab. 3. VISORGPT adopted GPT-2 (base) architecture and was trained from scratch. We use all datasets reported in Tab. 2 to train VISORGPT, and Open Images and Objects365 are not involved to train VISORGPT†. In evaluation, each category is at least involved in  $\sim 80$  valid predicted sequences by prompting (§ 3.2).
Table 4: Evaluation on training corpus scale and prompt templates of VISORGPT. The similarity between real probabilistic prior and the learned one is measured by KL divergence (KL Div).
Models	Prompt	KL Div on COCO (↓)	KL Div on Open Images (↓)	KL Div on Objects365 (↓)
Location	Shape	Relation	Location	Shape	Relation	Location	Shape	Relation
VISORGPT†	Ta	1.133	1.483	0.452	-	-	-	-	-	-
VISORGPT†	Ta+Tb	1.032	1.446	0.445	-	-	-	-	-	-
VISORGPT	Ta	1.212	1.813	0.561	0.890	2.775	3.715	1.969	1.345	2.790
VISORGPT	Ta+Tb	1.583	1.710	0.581	1.007	2.782	3.888	1.995	1.377	2.765
Evaluation on Learned Visual Prior. In Tab. 4, we present the measured similarity between real probabilistic prior and the one learned by VIsORGPT on the validation sets of COCO, Open Images, and Objects365, using KL divergence. The prompt template  $\mathrm{T}_a$  and  $\mathrm{T}_a + \mathrm{T}_b$  in § 3.2, are
Table 5: Evaluation on customized outputs  $(\%)$
Datasets	Quality (↑)	Controllability (↑)
Format	Matching	Size	#Instances
COCO	100.0	100.0	92.02	100.0
Open Images	99.97	99.40	89.35	98.71
Objects365	99.99	99.94	91.52	99.78
used for comparison. Overall, VISORGPT  $\mathrm{T}_a$  and  $\mathrm{T}_a + \mathrm{T}_b$  exhibit comparable performance, indicating both prompt templates have comparable capability for learning visual prior.
Evaluation on Customized Sequences. We present the quality of generated sequences and the performance of VISORGPT's controllability in Tab. 5. It is obvious that nearly all predicted sequences can be decoded successfully in three datasets. Additionally, in over  $99\%$  of sequences, all instances can match their respective locations. Besides, the table shows that VISORGPT achieves accuracies of  $92.02\%$ ,  $89.35\%$ , and  $91.52\%$  in controlling the average object size on COCO, Open Images, and Objects365 datasets, respectively. Furthermore, VISORGPT can achieve an accuracy of over  $98\%$  in controlling the number of instances across all three datasets. These findings demonstrate the strong capacity of VISORGPT in reasoning high-quality sequences and control the object size and number of instances in the scene.