Tabular Data to Image Generation: Benchmark Data, Approaches, and Evaluation
Authors: Alex Tang
Gromit Yeuk-Yin Chan
Ryan A. Rossi
Chang Xiao
Eunyee Koh
Venue: Submitted to ICLR 2023 (2023)

Abstract: In this work, we study the problem of generating a set of images from an arbitrary tabular dataset. The set of generated images provides an intuitive visual summary of the tabular data that can be quickly and easily communicated and understood by the user.
More specifically, we formally introduce this new dataset to image generation task and discuss a few motivating applications including exploratory data analysis and understanding customer segments for creating better marketing campaigns. 
We then curate a benchmark dataset for training such models, which we release publicly for others to use and develop new models for other important applications of interest.
Further, we describe a general and flexible framework that serves as a fundamental basis for studying and developing models for this new task of generating images from tabular data.
From the framework, we propose a few different approaches with varying levels of complexity and tradeoffs.
One such approach leverages both numerical and textual data as the input to our image generation pipeline. 
The pipeline consists of an image decoder and a conditional auto-regressive sequence generation model which also includes a pre-trained tabular representation in the input layer.
We evaluate the performance of these approaches through several quantitative metrics (FID for image quality and LPIPS scores for image diversity).

Under review as a conference paper at ICLR 2023
TABULAR DATA TO IMAGE GENERATION: BENCHMARK DATA, APPROACHES, AND EVALUATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we study the problem of generating a set of images from an arbitrary tabular dataset. The set of generated images provides intuitive visual summary of the tabular data that can be quickly and easily communicated and understood by the user. More specifically, we formally introduce this new dataset image generation task and discuss a few motivating applications including exploratory data analysis and understanding customer segments for creating better marketing campaigns. We then curate a benchmark dataset for training such models, which we release publicly for others to use and develop new models for other important applications of interest. Further, we describe a general and flexible framework that serves as fundamental basis for studying and developing models for this new task of generating images from tabular data. From the framework, a few different approaches with varying levels of complexity and tradeoffs. One such approach leverages both numerical and textual data the input to our image generation pipeline. The pipeline consists of an image decoder and a conditional auto-regressive sequence generation model which also includes pre-trained tabular representation in the input layer. We evaluate the performance of these approaches through several quantitative metrics (FID for image quality and LPIPS scores for image diversity).
propose
INTRODUCTION
In recent years, conditional image generation has been one of the most important directions in the line of research for generative models, due to both its technical challenges and numerous potential applications of such technology. However, most of the works that investigates conditional generation of images consider images (Gatys et al., 2016) or text (Ramesh et al., 2021; 2022) as input. Based on this fact, it is natural to ask whether we can extend the input to other data types to discover the potential benefit of image generation models to a larger variety of domains. Hence, in this work, we study the possibility of generating images from given tabular data, motivated by its promising capability to be applied in customer segmentation analysis for marketers and exploratory data analysis.
More specifically, we consider the following problem. Given a (tabular) dataset, or more generally a subset of rows and columns of the dataset', how images that describe it? Additionally, the set of images generated from the dataset should characterize the key trends, patterns, and segments (clusters) in the data. Such image generation model would yield interesting possibilities. For instance, suppose we have a dataset of customers and the items they purchased, then instead of performing thorough data mining that requires intensive technical expertise, an image that illustrates specific segment of customers purchasing specific items already reveals valuable information about the underlying traits of consumer behaviors, which can be easily used in future targeted marketing campaigns. Hence, tabular-data-to-imag generation can likely bring up interesting usage.
automatically generate set of high quality
Tabular data to image generation has many important and practical applications. One important application is as a fundamental tool for exploratory data analysis. Consider a user that is interactively
For convenience, the term dataset is used to refer to subset of rows and columns from a dataset well as the full datasct.

---

Under review as a conference paper at ICLR 2023
exploring large dataset of interest using standard visual analytics platform such as Tableau. In Figure 1, we provide an intuitive example demonstrating the utility of this problem for real-time interactive data exploratory and understanding. In this example, the user is exploring large dataset using an interactive scatter plot matrix where they can select the data points of interest to explore by simply brushing over them. The user draws rectangle around the data points (rows) and attributes (columns) of interest, which intuitively selects all such rows that satisfy the user-defined constraint. This selection then triggers a query our data table image generation model, resulting in the top-3 images being generated that are shown on the right in Figure 1. Most importantly, the user can immediately understand what the data points they selected actually represent by simply examining the intuitive set of images generated from the tabular data selected. They can then use the interface to select other groups of points representing a subset of rows the tabular data, and immediately understand the essence of what they represent.
data table
3.50
0..3
LIMII
11I
11
generated images
Data Table to Image Generation Model
User interactively selects rows and columns interest via brushing
Figure 1: Tabular Dataset to Image Generation for Interactive Exploratory Data Analysis. See text for detailed discussion.
In this work, we address the problem of sabularddtta-to-image generation by analyzing multiple possible approaches. To evaluate the performance of each of these approaches, we construct benchmark dataset consisting of 300 tabular-dats data and images that are paired up based on a simple mapping rule. The possible approaches include directly combining existing implementations of DALL-E and GPT-3, and an end-to-end approach, in which we train model similar to the architecture of DALL-E. We measure the performance of these approaches by computing the FID and LPIPS of the generated images with respect to the benchmark dataset, which captures the generated images' quality and diversity respectively.
The experiment results show that the approaches combining pre-existing models tend to generate images with better quality; however in terms of diversity, the end-to-end model we trained yields better performance. Nevertheless, the qualitative results indicate that all proposed approaches still have significant room for improvement, and it is worth investigating if a better performance metric should be employed.
This work makes the following key contributions. First, formally introduce the problem of image generation conditioned on tabular data input, which to the best of our knowledge has not been considered by any related works before. Second, we curate benchmark dataset for training models specifically for the new problem studied in this work. Third, we propose and develop a few different approaches for solving it, and extensively evaluate them the quality and diversity of the images generated from such models. Furthermore, the framework proposed in this work is highly flexible, giving rise a variety of other models that can be investigated in the future.
Summary of main contributions The key contributions of this work are as follows:
Tabular Dataset to Image Generation Problem: To the best of our knowledge, this work is the first to study the problem of generating images from tabular dataset. Furthermore,
2

---

Under review as a conference paper at ICLR 2023
also discuss number of important applications for interactive exploratory data analysis and to better understand segments in an online marketing application.
Benchmark dataset: To study this new and important task, we curated a benchmark dataset for model training and release it to the public for others to study this new task and develop better models for table to image generation, as well as to leverage the trained models for other important and novel applications in practice.
F Framework and Approaches: We present general framework for studying and developing approaches to solve the proposed data table image generation task. Furthermore, we propose and evaluate few different approaches for solving this task and extensively evaluate the quality and diversity of the generated images from such models across a range of different categories in our benchmark dataset.
Experiments are designed to extensively evaluate the proposed models for variety of data tasks. The proposed models along with the images generated are evaluated based on image quality and diversity of the images recommended.
Evaluation:
2 PROBLEM FORMULATION
We aim to generate images describing the underlying segment/trend of the given table. To formulate this mathematically, let L C Σ" be a language over some finite alphabet Σ (e.g. think Σ consisting of all ASCII symbols) and let _ be tabular input with rows and columns over the language C. For simplicity, we also mandate the first row of y. Y1: to always be the column names.
Our goal can hence be described as, given tabular input y E Lm we aim to generate an informative and high-quality image E E Nhxuxx, where h, w, d is the height, width and depth that resembles real- world images and contains relevant information about y.
In order to achieve our goal, one possible approach is to assume that the real-world images x and the tables y follow a joint distribution p(x: y), therefore given table y, we can model the mapping rule > x by the conditional distribution p(xrly), which can be learned by constructing a training dataset (0i, .x:) } After training on the dataset, we thus obtain a table-to-image model h: + approximating the true conditional distribution, and we can then use it to infer set of images for any new unseen tabular dataset Vnew as = h(ynew).. ). We will implement and analyze this approach in detail in the later sections.
To evaluate the performance of approaches that generate images given tabular input, we adopt common metrics that measure the quality and diversity of the images generated, namely FID scores Heusel et al. (2017) and LPIPS Zhang et al. (2018). However, despite methods for quantifying the correlations between text and images has been proposed recently (e.g. CLIP (Radford et al., 2021)), is not obvious how the relevancy between tables and images should be measured, since merely considering the textual component within tabular data risks losing structural information contained in the table, which is in fact key component we aim to model during the image generation process in our work. Therefore, we simplify the connection between tables and images based on simple rule to be described in the next section, and inspect relevancy qualitatively based on the visual perception of the image.
BENCHMARK DATASET
We benchmark the performance of table-to-image generation models based on a dataset consisting of 300 pairs of table and images. All images are downloaded from the Internet based on search results using keywords of common groceries, e.g. meat or fish; and each of the tables contains a subset of records found in Kaggle's 's open dataset "Marketing Campaign" (Saldanha, 2020), in which we only include the columns that indicates the amount of sweet, meat, fish, fruit, gold and wine purchased within transaction by a consumer. The relevant columns are then standardized based their column- wise mean and standard deviation, and we limit the resulting float number representation to decimal points.
Next, we employ the idea of weak supervision (Ratner et al., 2016) in order to construct the dataset of table-image pairs. Weak supervision is labeling technique that creates data-label pair by
3

---

Under review as a conference paper at ICLR 2023
assigning labels based on simplified mapping rule that can be done efficiently and automatically, instead of establishing the mapping manually, which can often be expensive and hard to obtain. Although the idea of weak supervision is first proposed to overcome the difficulty when dealing with insufficient training data for classification tasks, in this work we explore its possible application in image generation tasks and adopt its idea in constructing our benchmark dataset.
Hence, under this paradigm, we identify simple relations between a given table and an image based on the following rule: for table y E _ whose rows are sampled from the aforementioned Kaggle dataset and normalized column-wise, we assume the first row consists of column names and (3iy) E E for all E {2.... m) and j E ...,..,,, where the operator (+) + { extracts the numerical value of a string in the language L if the string represents a valid decimal number, otherwise returns We then associate an image produced by a keyword yıj with the table vk E
m), Mkj arg (1. (Ukz) and define the column j as the pivol column of table y: in other words, we say column is the pivot column if for every row k, (yk,) is greater than that of any other columns in the same row, and an image produced using the pivot column name as the keyword is paired up with y.
With this procedure, we defined 300 table-image pairs across categories: meat, wine, sweet, fish, gold, fruit, each with 50 table-image pairs. All images are resized to 256 x 256, and all tables consist of 20 rows.
APPROACH
In this section, we formally describe the proposed approaches for our new problem studied in this work. We discuss the advantages of each and begin by describing the simplest first. Within each subsections, will also discuss simple variations of each methods and their possible improvements. Since generating images based on tabular data is a new problem that has not been investigated before, we start with straightforward composition of pre-existing models gain better understanding of the key technical challenges for table-ime image generation, as well as setting baseline for comparing the strengths and weakness of different approaches. We will also investigate the possibility of directly designing an end-to-end approach for table-image generation, and compare the results quantitatively and qualitatively against the simpler composite approaches.
4.1 DALLE-COLUMNIN COLUMNNAME
For given table y E L generated based on the process described before, we directly input the pivot column names to DallE (Ramesh et al., 2021; 2022) and select the top images generated based on their CLIP score with respect to the input column name. For example, given table whose pivot column name is "MutFruits", we will directly input "MntFruits" into DallE to generate images.
4.2 DALLE-GPT3-CAPTION
Similar to the previous approach, we also pick the pivot column name from the input table. However, instead of directly feeding it into Dalle, we first input the pivot column name to GPT-3 (Brown et al., 2020) along with textual description asking it to generate suitable text prompt, and after retrieving its reply, we use it as the input to Dalle and finally select the top images with the highest CLIP score. For instance, with table whose pivot column name is "MntWines" we input the following text to GPT-3: Generate an image caption using "MnrWines." " The GPT-3 model will then sample reply from the underlying conditional distribution, say, A bottle of wine on a mountaintop, with the caption "MntWines: The best wine for your next adventure." We will then use its reply directly as the input to dalle-mini for image generation.
4.3 HALLE-TABBIE
Given the above approaches, it is natural to ask whether we can design end-to-end process for the image generation process (Figure 2). Hence, based on DALL-E and DALLE-minis model architecture (Ramesh et al., 2021), we design a two components: a VQGAN image encoder/decoder (Esser et al., 2021) and a BART sequence- to-sequence model (Lewis et al., 2019) that outputs image tokens auto-regressively given tabular
generation model based on
4

---

Under review as a conference paper at ICLR 2023
representations as inputs. We mathematically introduce the objective functions model is trained to optimize in the following paragraphs.
Same as we have defined above, let _ be a tabular input and f N be generated RGB image with height h, width w and color depth d. We also define x E Nhxwxx as real-world RGB image and E Z the latent representation for image. Hence, assuming the data generation process can be modeled by the Markov chain I + 2 - we can lower-bound the oggliikeihood of T. y, p(z, y) by
p(z, U. 2,, = log / pla))(aloo)p(ls)dd= log Ex1= (00l=:(()pp() ((x)p(ul=)) =EZ
p(z, v) = log
=EZ
2 Eg æ
log log ((gp())
= BBg logp(*) +
- (ogp(=/))) +og + logp(() log p(z)
Eg log [ogp(æ| - (ra((<//))llll)
= (xlar) log (x(l2) DKL (el(ee@el=)llll<eeeeee
where DKL. the KL-divergence between two probability distributions, qys Pa the encoder and decoder of VQGAN respectively, and P& is the BART decoder. The last line in the objective function above is commonly known as ELBO in variational inference literature, and in our model we approximate those distributions through common neural networks. However, in our problem setup we have multiple distributions to optimize, therefore we follow similar training approach to that of DALL-E mini's by splitting the training process into stages, which describe in detail below.
1. Fine-tuning VQGAN Same as previous works, we started with the VQGAN model trained on ImageNet (Deng al., 2009). We fine-tuned our VQGAN model for 10 epochs to adapt it to our image dataset's 's distribution.
Fine-tuning TABBIE Since our model crucially relies on tabular data as input, we need to produce good representation for tables. In this work, utilize TABBIE (lida et al., 2021) in model to extract relevant structural and textual/numerical information for the input table. TABBIE learns tabular representations through objective known corrupt cell detection, which asks the model to detect if any element within the table was replaced by an inconsistent one. The embeddings produced by TABBIE has shown perform well on various tabular data mining tasks, therefore utilized the pretrained representation as input to the BART encoder. To ensure the embedding vectors align with the tables we have, we fine-tuned TABBIE based on the default configurations.
Transforme generates image representation besed on input tabular (seq2seq)
Pretrained tabular representation
Cluster Lables
Clustered data as input
regressivee (cecoder-only) transfomer
1010 1010
Pretrained image representation (eg. VOGAN)
Pair taples real-world images week supervision
Figure 2: Overview of the end-to-end training process. See for detailed discussion.
5

---

Under review as a conference paper at ICLR 2023
3. Training BART Finally, based on the training procedure of DALL-E mini, we followed similar approach for training the transformer. We preprocessed the training data by first encoding the images into tokens using VQGAN, and paired them up with their corresponding table's embedding vectors. The tabular embeddings are then passed into BART, aiming to produce an output sequence that matches the corresponding image. We trained the model for 30 epochs until the validation loss begins to deviate from the training loss. We also performed a grid search for the hyperparameters used in sampling from the conditional distribution for image generation by maximizing the CLIP score of the generated images with respect to their pivot column names.
4.4 DISCUSSION
It is also possible to generate images following similar methodology when textual input is given, this includes directly inputting the table into Dalle-mini, asking GPT-3 to generate caption given the tabular input, and a more comprehensive approach that trains a tabular-image image joint representation similar to CLIP. However, how define good paring rule between a table and an image remains the main challenge for producing a model that generates meaningful images.
Aside from the difficulty of data preparation and bechmarking. since we have only focused on dealing with tables that mostly contain numerical information, it is also worth investigating for the end-to-end model approach, whether a latent representation, especially those trained based on tokenization the input first actually encodes those numerical trends. In fact, it will also be interesting if we can directly use the numerical values within the table as its embedding vectors, given that we have identified a reasonable method to ensure the consistency of the embedding dimensions.
However, as we mentioned above, we currently limit the input table to only contain numerical data, and it is still unclear what we should expect if other data type are also present in the input table. Furthermore, even for our current task, is also worth asking if we can do anything beyond merely generating an image of an item, such as including persons or even backgrounds that also reflects some information about the input table. We leave these possible improvements as part of our future directions.
5
RELATED WORK
5.1 TEXT-BASED IMAGE GENERATION
Recently, there has been a lot of work towards generating images from textual prompts. Some of the most famous results are those of DALLE Ramesh al. (2021) and DALLE-2 Ramesh et al. (2022). The model architecture of DALLE consists of an image encoder/decoder and a transformer, and learns generate images by concatenating the textual input token along with the image tokens by modeling it as a sequence-to-sequene transformers to produce an image as the output sequence. On the other hand, DALLE-2 leverages CLIP embeddings that jointly model text/image pairs for image generation. There are also other works (Saharia al., 2022; Rombach et al., 2022; Nichol al., 2021) that investigates text-to-image generation methods. However, all of these works have focused on generating images from text-based prompts whereas we focus on image generation from numeric tabular data.
task, thus taking advantage of the powerful capabilities of
5.2 TABULAR DATA REPRESENTATION LEARNING
There has been lot of recent work focused on developing models for learning representations from tabular data to perform various downstream tasks, including trend detection and tabular question- answering. We highlight a recent approach known as TABBIE (Tida et al., 2021) that learns the embedding of tables by training on the objective of detecting corrupted cells. One notable results of the embdiing learned by TABBIE is that for identifying numeric trends in the input table, outperforms multiple previous works and demonstrates the potential of applying these embeddings to more complex tasks. Nonetheless, all of these approaches have not focused on our specific task of generating image set of images from given tabular dataset.

---

Under review as a conference paper at ICLR 2023
To the best of our knowledge, this paper is the first to introduce this task, curate benchmark dataset for it, develop techniques for solving it, and evaluating the approaches both in terms of quality and diversity of the images generated by them from the tabular datasets.
Table Image quality (FID) and diversity (LPIPS) results.
Data
Model
Image Quality (FID) | Image Diversity (LPIPS) t
415.29 + 18.18
0,75 + 0.05
Gold
DALLE-GPT3-Cappe
335.93 + 16.53
0.72 + 0.01 0.77 + 0.06
DALLe-Tabbie
701.04 + 29.43
DALLE-ColumnName
295.94 + 15.22
0.77 + 0.04
Wine
MALLE-GPT3-Cappie
277.66 + 19.42
0.79 0.03
DALLE-Tabbie
338.90 + 33.54
0.86 + 0.07
HALLE-Column..
492.55 + 33.32
0.78 = 0.03
Meat
WALLE-GPT3-Captinn
358.04 + 29.98
0.72 = 0.05
DALLE-Tabbie
515.35 + 46.32
0.81 + 0.06
DALLE-Column
354.21 + 20.61
0.82 = 0.03
Fruit
DALLE-GPT3-CAN
426.27 + 22.28
0.79 = 0.02
DALLE-Tabbie
359.18 + 38.66
0.78 0.05
DALLE- NaaLe-ColumnName
327.84 + 17.10
0.78 + 0.04
Sweet
DALLE-GPT'S GAAEE--GPP--atttoon
465.79 + 23.07
0.70 + 0.04
DALLE-Tabbie
437.07 + 43.15
0.83 = 0.07
DALLE-ColumnNanc
506.60 + 12.57
0.73 + 0.02
Fish
DALLE- DALLE-GPT3-CCE GPT3-Caption
509.78 = 13.66
0.74 = 0.03
DALLE-Tabbie
530.37 + 26.70
0.78 + 0.06
EXPERIMENTS
To understand the utility and effectiveness of the proposed approaches for this new problem, we designed experiments quantitatively evaluate the quality and diversity of the images generated from the various approaches. Furthermore, we also show the actual images generated from the proposed models and provide detailed discussion on some of the interesting insights observed.
6.1 EXPERIMENTAL SETUP
For each approach, we compute its FID with respect to the benchmark dataset as follows: for DALLE- ColumnName and DALLE-GPT3-Capion we randomly partition the 20 images into groups, each with 5 images, and we measure the FID between the images within each partition against all images in the benchmark dataset under the same category; and for DALLE-Tabbie, we also randomly partition the 200 images into 4 groups and measure their FID with respect to the benchmark dataset. Note that since observed the variance of DALLE-Tabbie to be typically larger than the other approaches, we reported results of 200 generated images. Nevertheless, similar performance was observed when using only 20 images well. To measure LPIPS (Zhang et al., 201 8), simply pair up all the images produced by each approach with those in the benchmark dataset and compute the mean and variance of the resulting scores.
6.2 QUANTITATIVE RESULTS
quantitatively compare the quality and diversity of the images generated by the proposed models. For this, we the FID (Heusel et al., 2017) evaluation metric to quantify the quality of the images and leverage LPIPS (Zhang et al., 2018) to evaluate the diversity of the images generated from the proposed models. Results for various data in our benchmark corpus reported in Table 1. We observe that the model generating the highest quality images highly depends on the underlying data category from the benchmark corpus. Notably, DALLE-GPT3--Capion performed best for Wine,
7

---

Under review as a conference paper at ICLR 2023
(b) DALLE-GPT3-Caption
(c) DALLE-Tabbe
Figure 3: Qualitative comparison of the images generated from the different approaches.
Meat, and Gold in terms of the quality of images generated whereas MaLLE--oolunNaeee generated the highest quality images for Fruit, Sweets, and Fish. Intuitively, Wine and Meat are often purchased together, and thus this makes sense. Conversely, fruits and sweets are also related. Furthermore, in nearly all cases, DALLE-Tabbie generates the images with largest diversity. The only exception is for the fruit dataset where HALLE-ColumnNamt achieves the best diversity. In this case, DALLE-Tabbic generates images of very high quality, outperforming DALLE-GPT3-Cape to the quality of HALLE-ColumnNee Hence, we observe a trade-off between image quality and diversity.
and relatively close
DALLE-ColumnNNan Comu
B
DALLE-GPT3-Caption
c
DALLE-Tabbie
Figure 4: Comparison of the diversity of the images generated by the proposed approaches.
6.3 QUALITATIVE RESULTS
We now compare the images generated from the approaches qualitatively in Figure 3. particular, show images from the fish data generated from each of the approaches described in Section 4. Interestingly, we see that GALLE-GPT3 generates a painting-like image person fishing on cloudy day with mountains behind them. However, DALLFFCClummNamm generates realistic
8

---

Under review as a conference paper at ICLR 2023
image of a fish whereas DALLE- generates image seemingly in-between these two extremes in the sense that the fish is realistic looking with some artistic features such as the dorsal fin of the fish near the top of the generated image in Figure 3. Furthermore, the image genertaed by DALiE-Tabbie also contains two fish facing in opposite directions. In Figure 4, we investigate the diversity of the images generated by the proposed approaches. Notably, the images generated by DALLE-ColumnName are all very similar showing fish with mostly the same features. Similarly, DALLE-GPT3-Caption generates images with very similar scene that includes an individual fishing with mountains in the background. All of the images have very similar colors as well, Both DALLE- GEEEE---T3--piiio suffer in terms of the diversity of images that these models
ColumnName and are capable of generating. However, DALLE-Tabbie is able to generate images that are significantly more diverse as shown Figure 4. Notably, the images are even diverse in terms of style, that is, realistic images as well as those that appear to be drawn, to even those that are a hybrid with some realistic and artistic features and colors.
7 CONCLUSION
In this work, we considered the problem of aabula-ddata-to-imaa generation motivated by various real world applications and investigated multiple potential approaches to achieve this task, and to the best of our knowledge this work is the first to study such problem. We created benchmark dataset in order to evaluate the performance of different methods and construct an end-to-end model for image generation. We identified framework for developing approaches solve the proposed problem, and we evaluated the performance of those approaches quantitatively and qualitatively.
We conclude that future work should develop more sophisticated approaches for this new task and explore using such techniques for other exciting applications. Furthermore, by releasing an initial benchmark dataset for this important new learning task in order facilitate others to work on this problem and develop new approaches to solve it, we hope that future work will expand and enhance this initial benchmark dataset with other types of tabular datasets and corresponding image pairs.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 100,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 12873-12883, 2021.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolu- tional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit lyyer. Tabbie: Pretrained representations of tabular data. arXiv preprint arXiv:2105.02588 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to -o--nnce-----uuuneeeeeee pre-training for natural language generation, translation, and comprehension. arXiv preprint urXiv:1910.1344 2019.

---

Under review as a conference paper at ICLR 2023
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115,, 2022.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 1.Xiv:2055.11487, 2022.
Rodolfo Saldanha. Marketing campaign (kaggle dataset), 2020.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, p 586-595, 2018.
10


Reviews:
[{"review_id": "rp6-5CXkzsh", "reviewer": "Reviewer_PnA1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on an interesting topic of generating images based on tabular data. It provides a benchmark dataset and proposes some approaches with varying levels of complexity and tradeoffs.\n", "summary_of_the_review": "The paper is interesting, but there are still some concerns. The authors may try to address the issues and make the paper clearer.\n", "strength_and_weaknesses": "The main idea of the paper is interesting. But the motivation what we could benefit from the generated images is not very clear. \n\nThere are some questions:\n1. The authors need to provide more examples when the generated images help. The reviewer is confusing that we can generate an image for each sample or only for a whole set of tabular data.\n2. The authors may consider providing more illustration examples in the paper. The current illustration images seem to be subjective.\n3. Does the ability of the method depend on the quality of the pre-trained model?\n4. In some real-world tasks, we may not know the exact meaning of the column, and it is difficult to find the pre-trained model in the specific area. For example, if the table is related to housing prices or climate change, could the authors show the corresponding images?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hdCAnSVBe-D", "reviewer": "Reviewer_CFxB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the problem of mapping from table data to images. The author's test strategies directly generate images from tabular data (which can include numerical values). This would give an end user the ability to create a visual summary of the data they are looking at. Generated images are evaluated using FID and LPIPS, and results for methods are additionally qualitatively discussed. ", "summary_of_the_review": "I think the idea is interesting from an application perspective, and the problem is very well defined by the authors- specifically the problem of generating a \"visual summary\" of table data. But throughout the paper, there is not enough detail on the data that the models are being trained on. Additionally,  there is not nearly enough qualitative output - which would be expected for a paper that aims to do conditional generation. When figures exist are generally hard to follow and sometimes actively confusing (ex: components of Figure 1). \n\nNo evaluation in the paper seems to critically address one big component of the paper - do the resulting images well summarise the table data in some meaningful way? While I understand this is very difficult to quantify, there is zero to minimal discussion on this point. Since this is a conditional generation problem, do my generated images in some way tell me about the data they are generated from? A good solution to this problem would not just generate images that are not only high quality, which your metrics measure, but also relevant to the data they are conditioned on, and I'm not sure that this is addressed in any detail.", "strength_and_weaknesses": "Strengths\n- [major] Problem formulation is very understandable\n- [major] New modeling approaches for end-to-end conditional generation \n\nWeaknesses\n- [major] Not nearly enough results and qualitative output, the paper is about the value of visually summarizing tabular data but there is minimal to no qualitative output \n- [major] Figures are out of place and are relatively difficult to follow, (ex: components are unrelated in figure 1 and are actively confusing)\n- [major] Figure from TABBIE paper seems to be directly copied and pasted onto one of your figures\n- [minor] Benchmark dataset is described, but very little information is given (ex: what size are the tables, distribution of categories)\n- [minor] usefulness/applications are very simply motivated and then rarely addressed when they are a big component of the problem.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZW6Z4amhT6S", "reviewer": "Reviewer_Leo6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the tabular-data-to-image generation problem, which is novel and somewhat interesting. To study this unexplored problem, this work first curates a benchmark dataset containing 300 pairs of tables and images. Then, this paper utilizes existing models, e.g., DALL-E, GPT3, VQGAN, and TABBIE, to generate images from the tabular data. Finally, the authors qualitatively and quantitatively evaluate the proposed method.", "summary_of_the_review": "Although this work addresses a novel and interesting tabular-data-to-image generation problem, the quality of writing is poor, the technical contributions are limited, and the experimental evaluations are weak. Based on my aforementioned concerns, I recommend rejecting this paper.", "strength_and_weaknesses": "### Strengths\n\n- This work studies a novel and somewhat interesting problem, i.e., tabular-data-to-image generation.\n- This paper curates a new benchmark dataset for the tabular-data-to-image generation problem.\n\n\n\n### Weaknesses\n\n- The quality of writing is poor. It is hard to follow this paper.\n- The technical contributions are limited. This work simply applies prior techniques, e.g., DALL-E, GPT3, VQGAN, and TABBIE, to the tabular-data-to-image generation, lacking novel technical designs.\n- The experimental evaluation is weak. The results in Section 6 are insufficient to support the main claim in the Abstract, that is, \"The set of generated images provides an intuitive visual summary of the tabular data\".\n- In Section 6.1 Qualitative Results, this paper mainly discusses the quality of generated images. However, from my point of view, the correlation between the tabular input and the generated images is more worthwhile to discuss.\n- The proposed benchmark only contains 300 pairs of tables and images with a limited number of categories. I don't think this is sufficient to train a generative model for real-world tabular-data-to-image generation problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject