Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel approach to generate synthetic tabular data using large language models. 
strength_and_weaknesses: Strengths:

The authors proposed an interesting way to use large language models. It's a novel use case.

Weaknesses:

1. The computation cost for the proposed approach seems profitably expensive. In table 6, both GReaT and Distill-GReaT required two order of magnitudes bigger compute budget comparing to other baselines. This leads to serious scaling concerns. 

2. Have the authors consider using few-shot evaluation from the large language models as the alternative way to generate synthetic tabular data? It would be cheaper than fine tuning such a big model using hundreds of epoches.  With proper prompting, large language models should be able to generate desired outputs.

3. I am not sure the significance of the proposed method. It's not clearly to me the significance of the deltas in those measures reported in the experiment section. 


clarity,_quality,_novelty_and_reproducibility: This paper is clearly written. The text, related work, figures and tables are well prepared. 
summary_of_the_review: The authors did find a novel application of large language models. However, it's unclear if the gains of the proposed method can justify the 100x increase in the computation cost.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
strength_and_weaknesses: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
clarity,_quality,_novelty_and_reproducibility: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
summary_of_the_review: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the use of large-scale autoregressive generative language models (LLMs) for generating tabular data. Several recent works have similarly attempted the task of generating tabular data using popular computer vision methods such as Variational Autoencoders and Generative Adversarial Networks, but require an initial preprocessing step of encoding the categorical columns and (optionally) transforming the numerical columns, which often have a large impact on the performance of the generative model. In this paper, a different approach is taken by using a transformer-decoder network, a popular architecture in the NLP field. In this approach, the columns are first converted to text using a textual encoding scheme, the feature order is permuted to allow arbitrary conditioning and sampling, and fine-tuning is performed using a pretrained generative LLM.
strength_and_weaknesses: Strengths:
- The proposed approach is easier to handle because it removes the bottleneck of the first preprocessing step for the numeric and categorical columns 
- The textual subject-predicate-subject encoding scheme, together with the random permutation of the feature order, provides the end user with full probabilistic control over the sampling procedure of the generative model
- The paper shows significant improvements over the state of the art evaluated with the proposed methods

Weakness:
- Lack of related works. The authors claim that there are no previous works using pre-trained LLM to generate tabular data. However, a quick Google search revealed that there are some recent works (2021) that have considered a similar architecture for generating tabular data, including time series data. Granted, the pre-processing steps are different and this work is new in that regard. Regardless, the authors should highlight such related work and differentiate their work or justify the omission 
- Bad samples. The authors state in Section 3.2 that some synthesised samples that violate the resulting patterns for tabular data have been discarded, but fail to provide an explanation of why such a violation occurs or provide examples of such samples to guide the reader's imagination
- Practicality: the proposed methods require significantly longer fine-tuning (~9hrs) compared to other baseline methods (< 2mins). The authors have not commented on this or justified why such a computationally intensive approach is preferable

clarity,_quality,_novelty_and_reproducibility: - The writing is clear, detailed and not ambiguous
- The textual encoding scheme proposed by the authors in quite novel in this setting
- The work seems easily reproducible, although the authors are strongly encouraged to share their code

Minor corrections: 
- Confused Indices. In section 3.1, definition 1. the authors seem to have confused the indices for rows and columns. m was originally defined for rows, but later used for columns/features. In general, the authors are advised to clarify the notations in the various definitions
- Typos. Section 4 (baseline, line 3 and DCR, line 5)

Further clarification needed:
- The authors claim that previously proposed methods such as mode-specific normalisation introduce artificial ordering. The authors should clarify what is meant by this statement
summary_of_the_review: The proposed method is a breath of fresh air compared to previous work on GANs and VAEs for generating tabular data. Transformer architectures are increasingly dominating various fields, and this approach seems promising for future deep-learning-based synthesis of tabular data, but how to protect the privacy of the data synthesised from such models still needs to be studied in depth.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes GReaT (Generation of Realistic Tabular Data), a large language model fine-tuned to generate samples for tabular datasets. First, GReaT encodes examples in a tabular dataset into sequences of text, then GReaT fine tunes a large pretrained language model using the textual representation of the tabular data. Once tuned, GReaT can generate entirely new examples, or sample missing values using arbitrary conditioning.

Experiments on 4 tabular datasets (3 classification and 1 regression) demonstrate GReaT generally outperforms existing tabular data generators on a wide and comprehensive range of evaluation metrics.
strength_and_weaknesses: Strengths
---
GReaT operates directly on the tabular dataset, avoiding lossy data preprocessing steps such as 1) encoding of categorical data (e.g., one-hot encodings); 2) data scaling or normalization; 3) missing value imputation; and 4) outlier removal. These data preprocessing methods can often result in loss of information and ultimately reduce the quality of generated samples.

GReaT is a simple and novel approach for generating samples for tabular datasets, the most common type of data used in industry. This is a potentially significant tool that can help combat noisy or missing values, or can be used to balance imbalanced datasets.

GReaT fine-tunes a large language model using random feature order permutation, allowing sampling with arbitrary conditioning. This also allows GReaT to leverage the vast amount of contextual knowledge learned during pretraining of the large language model to better generate appropriate values for certain features given the values of other features.

In comparison with existing methods, the experimental results suggest GReaT generates samples that 1) better substitute as training data, 2) are less discernible with the actual training and test data, and 3) qualitatively produce more appropriate values than existing methods.

Weaknesses
---
Experiments are run on just 4 rather small tabular datasets, limiting claims of GReaT's generalizability.

The datasets used contain a relatively small number of features; thus, I am unsure of how GReaT would perform for much higher-dimensional tabular datasets resulting in significantly longer input sequences for fine-tuning.

One of the motivations for GReaT is the fact that tabular datasets may contain sensitive or personal information, but wouldn't a high-quality data generator potentially reproduce some of this personal data since it is trained on that information?

Minor Weaknesses
---
What is the train-test split percentage mentioned in the "Data sets." paragraph in Section 4?

In Figure 3, the occupation "Adm-clerical" in the first row of the bottom-left table does not match the first input sequence of the textual representation where the occupation is "Transport-moving".

In the paragraph "Baselines." in Section 4, "...on a only..." -> "...on only a...".

Table 1 caption, "...in case of the regression..." -> "...in the case of regression...".

In the paragraph "Baselines." in Section 4, "...an variational autoencoder..." -> "...a variational autoencoder...".
clarity,_quality,_novelty_and_reproducibility: As far as I can tell, the work is certainly novel and is an interesting and clever application of large language models. Since the approach is very straightforward, and the code is provided as an easy-to-use Python package, the reproducibility of the results seems very likely.
summary_of_the_review: GReaT leverages large language models to generate samples for tabular data in a way that avoids many of the lossy data preprocessing steps required by existing approaches, resulting in high-quality contextually-relevant tabular examples. Overall, the number of datasets evaluated is rather low, however, the positives currently slightly outweigh the negatives.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a way to adapt pretrained large language models to generate tabular data. The approach comprises a pre-processing of inputs to convert parts of tabular data into strings, a post-processing of the output string into tabular data, a fine-tuning the LLM, and varying the sampling strategy. While none of these methods are technically novel, the results -- especially compared to GANs and VAEs -- are strong, and merit acceptance. 
justification_for_why_not_higher_score: While the results are strong, the techniques are not necessarily that novel, so an ICLR audience would likely be better served by another paper.
justification_for_why_not_lower_score: The method requires few tricks, the empirical results are strong, and the focus is on an underserved area.
Decision: Accept: poster