Venue: NeurIPS 2022 Workshop SyntheticData4ML

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this work, the authors present a few-shot data augmentation framework for dialogue understanding by prompting large pre-trained language models. They further introduce a weakly supervised labeler to filter prospective data points. Models fine-tuned on their augmented data approach or surpass existing state-of-the-art performance on DAILYDIALOG and FACEBOOK MULTILINGUAL TASK-ORIENTED
DIALOGUE datasets.
review_title: Few-shot data augmentation for dialogue understanding

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
This paper proposes a filtering technique to select the best quality synthetic data generated by prompts for the dialogue task. They evaluate various augmentation methods on the classification tasks in DAILYDIALOG and the intent detection task of FBTOD.
What the paper addresses an existing problem in dialogue augmentation, however whether it is the correct approach or not is entirely clear to me (see question 1).

##########################################################################

Positive points about this paper

Addressing the problem of out-of-distribution data generation is an interesting idea. Augmentation is a valid approach to address this lack of data in the dialogue task. 
The paper covers the literature well and the experiments showcase the usability of the approach. It is well-written and experiments are clearly defined for the most part. 

##########################################################################

Concerns and questions:

- My main concern for this paper is as follows: The authors say that prompted augmentations are less potent because they lack task-knowledge, which can be distilled from ground-truth (“gold”) samples by training an auxiliary model. They then use that model to filter out inconsistent generated utterances. The filtering model would work with the task knowledge ONLY if there is already relevant and in-distribution data generated by the LM. It doesn't necessarily address the problem of lack of task-knowledge in the augmentation process. Isn't that the main problem this paper is trying to address?

- What is the impact of the prompt you used on the results? How much effort went into engineering the prompt? Did you try different variations or did you not "finetune" the prompt?





review_title: Interesting problem to tackle, but doubtful about the motivation of the proposed approach

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Interesting idea to use a classifier post inference in the feedback loop. I found it to be creative, and apparently easy to use as a plug-in method. The consistent improvement across accuracy and F1 scores for the Spanish dataset is also attractive.
review_title: Interesting approach that demonstrates promise

Decision: Accept