Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose in this paper to use synthetic data for model evaluation and to create an automated suite of synthetic data generators (called SYNG4ME). SYNG4ME, in particular, has two main properties: 1) it provides reliable granular evaluation, and 2) it quantifies model sensitivity to distributional shifts. The paper discusses various SYNG4ME cases, which will provide some insights into using synthetic data for model evaluation.
strength_and_weaknesses: Strength :

1. The problem studied in this paper, i.e., using synthetic data for model evaluation, is interesting and important.

2. An adequate discussion of SYNG4ME use cases is provided, providing some insights into how to use synthetic data for model evaluation.

3. The entire paper is simple to understand. The authors thoroughly review all related work, summarising its benefits and drawbacks.

Weaknesses:

1. Although the problem studied in this paper is intriguing, it is not a novel one that has received much attention in the machine learning and computer vision communities. Using synthetic data to evaluate models is also not a new topic. On the other hand, the technology employed in the paper is not particularly novel.

2. As stated in the paper, SYNG4ME is only capable of handling tabular data. I believe its application scope is too narrow.
clarity,_quality,_novelty_and_reproducibility: 
Clarity and Quality: The paper is clearly written.

Novelty: The problem studied in this paper is interesting, but it has been widely discussed in the machine learning and computer vision community. Meanwhile, using synthetic data for model evaluation also is not a new topic, and the technology used in the paper is not very innovative.

Reproducibility: I think this paper is easy to reproduce.

summary_of_the_review: This paper may provide some insights into using synthetic data for model evolution, but I believe it will have little impact on the community (the novelty is limited).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for using synthetic data to test predictive models for their performance on sparsely populated subgroups and on robustness under distributional shift. The framework focuses on tabular data. 
strength_and_weaknesses: Strengths:

Leveraging synthetic data for AI testing is timely and critical for many tasks, which is the focus of this work.

The paper is clearly written and easy to follow. 

The paper considers two scenarios – (1) when the original data suffers from sparse population of important subgroups and (2) when there is a distributional shift between training data and test data. 


Weaknesses:
The idea of using synthetic data to evaluate ML models is not entirely new. "Synthetic Data for Social Good" arXiv:1710.08874 has proposed  DataSynthesizer, a privacy-preserving synthetic data generator for creating pathological dataset in tabular domain to be used for model testing.   AITEST described in “Data Synthesis for Testing Black-Box Machine Learning Models” (arXiv:2111.02161) also considers group fairness testing as studied in this paper and leverages goal-oriented synthetic data generation in tabular domain for model testing. AITEST has used data constraint-based data synthesis, in addition to using CTGAN and TVAE as baselines, whereas the current work mainly uses CTGAN. Syng4me considers distribution drift for robustness testing whereas AITEST studies adversarial robustness. 

clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: OK
Novelty: low
Reproducibility: low (no code shared)
summary_of_the_review: While the paper pursues an important direction, the framework, the methods and the results presented do not provide enough novel insights on using synthetic data for testing fidelity, utility, fairness, and privacy of downstream ML models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an evaluation suite, which generates synthetic test set that helps to better evaluate the performance of supervised learning models for small subgroups (minority groups) and distributional shifts. Augmenting and creating synthetic data enable us to increase the number of data in small subgroups which help to decrease the variance resulting from evaluation metric's performance. Also adding shift to the test distribution make the model more reliable to be tested on different domain shiftings that would happen by testing in different test sets.
strength_and_weaknesses: The issues behind the evaluation on small test set is written clearly. The thorough experiments and analysis in the paper and appendix make the idea more convincing. 

However, the actual use-cases of such evaluations specifically for small subgroups is not very clear. Is this type of evaluation on synthetic test set is good if we want to get the performance of the metric on small subgroups separately? Since according to table 1 on majority samples the metric tested on D_test,f  is more reliable. Therefore some explanations on why do we want to evaluate on small minorities is beneficial.

In the proposed approach to generate synthetic test set, the generator is trained on a small test set. What is the effect of such a small test set that is used as training dataset on the generator's performance, the newly generated synthetic data and subsequently the performance of the evaluation metric. In other words, from low number of test set how much the generative model is able to learn the manifold from D_test,f.

It is not clear that how much the proposed distributional shiftings in generated synthetic test set are close to the shiftings that could happen during testing.

Similar to section 5.2.1 incorporating some examples in real-word test cases in 5.2.2 also can be very useful.




clarity,_quality,_novelty_and_reproducibility: The motivation, problem statement and the idea of using synthetic data for test sets looks novel and quite interesting. The positive impact of such evaluation suit on measuring the reliability and robustness of the evaluation metric can be very beneficial to the research community. Overall, paper is written quite neat however some further explanations can transfer the overall idea and purpose of the paper easier. 
summary_of_the_review: This paper and the idea of generating synthetic data and leveraging them for testing classification model can be promising in identifying  reliable metrics. However, there are some weaknesses that mostly are coming from not clear descriptions which should be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposed paper aims to introduce a framework for validating neural networks with synthetic data. Specifically, the discussed method proposes to train a generative model trained on the commonly used test data to generate a larger and more diverse synthetic dataset. The generated synthetic data is then used to more carefully validate a downstream task model. The argument of the paper is that validating on synthetic data can help lowering the variance of common real validation datasets and to help conditioning generative models to better understand how these models behave w.r.t. different target settings. The paper suggests that validating with synthetic data leads to more accurate performance estimates compared to using real test data alone. 
strength_and_weaknesses: Strengths:

- The paper addressees an important and interesting problem. 
- The reasoning behind the proposed method seems technically sound and the experiments toward granular evaluation and distributional shifts seem reasonable. 
- The paper is well-written and easy to follow. 
- Limitations of the method are discussed an an appropriate and fair manner. 

Weaknesses: 

- The paper only focuses on tabular data for which is rather easy to synthetzise data (compare to text and images). 
clarity,_quality,_novelty_and_reproducibility: The paper is very well-written, easy to follow, and the topic it addresses is important and interesting. Using synthetic data to validate machine learning models more carefully and w.r.t. distributional shits seems to be an important direction. I much appreciate that the paper provides a candid discussion toward the limitations and the scope of the proposed approach. 

The fact that the method currently only works on tabular data indeed is a strong limitation. Given that there exists severe differences between real and synthetic data in other domains (e.g. for images), it is not clear that the method could easily extend to these areas. The paper also discusses the limitation that generative models for synthetic data may fall short in approximating the real data appropriately. Towards this topic I would appreciate a more in-depth discussion compared to what is currently provided. E.g. I would appreciate pointers to existing work that explores how well generative models are able to approximate tabular data. 

Given that the paper introduces a framework rather than an algorithm, enough information is provided to replicate the proposed approach. 
summary_of_the_review: Overall, this is an interesting paper that can be accepted to ICLR 2023. The topic this paper address is important and novel, the paper very well written, and the conducted experiments are sound. Furthermore, the paper carefully discusses the scope and limitations of the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose SYNG4ME: a method of evaluating models by leveraging synthetic datasets. This idea is not new and there exists much prior work on this type of evaluation. What is novel is the focus on tabular data, which is comparatively less studied compared to domains such as images, text, and speech. The reviewers, however, found this limited scope more a weakness of the paper rather than a strength. Furthermore, the authors propose the empirical evaluation only on relatively simple datasets. Finally, the limited technical novelty was something mentioned in a number of reviews.

Extensions to other modalities, and evaluation on larger datasets would greatly improve the paper.
justification_for_why_not_higher_score: The limited novelty of paper, the focus only on tabular data, and the empirical evaluation on small datasets preclude me from justifying a higher score.
Decision: Reject