Exploring Synthetic Data Generation Techniques for Employment Type Classification in Job Advertisements
Authors: Kai Krüger


Reviewer paper summaries: The paper explores synthetic data generation (SDG) techniques for employment type classification in online job advertisements (OJAs) with the help of LLMs. It evaluates four SDG methods—plain prompting, sampling, precise attributes, and adjective attributes—and assesses their impact on the performance of classification models. The study also explores the use of Natural Language Inference (NLI) filtering to enhance data quality. Results indicate that models trained on synthetic data can effectively classify real-world job ads, achieving competitive performance. However, the study also notes volatility in outcomes, suggesting that further investigation is needed to fully understand these variations.
Summary of strengths: -The paper provides comprehensive experiments that are well described and documented in the Appendix.
-It evaluates multiple methods of SDG and investigates various aspects like data diversity, label-fit, and the impact of dataset size.
-A contribution is that the resources datasets, models and codebase) will be made publicly available, which can foster further research in this area.
-The study provides insights into synthetic data generation, which is particularly relevant in the context of low-resource frameworks.
-Despite high volatility, achieving good performance (e.g. up to 96% F1 score on real-world datasets) highlights the potential of synthetic data in improving model performance.
Summary of weaknesses: -While the authors mention that they make their datasets and models publicly available, I was not able to find a link to access these resources, difficulting reproducibility and understanding of the details of the paper.
-The study highlights the variability in diversity metrics but does not propose a unified approach to measure diversity effectively.
-The significant volatility in performance across different methods and dataset sizes is a concern, indicating potential underlying issues not fully addressed in the paper.
-It seems like to incorporation of randomness in the generation of synthetic data introduces uncertainty over the results. At what point this could account for inconsistencies is not clearly discussed in the paper. A possibility would be to use a deterministic decoding strategy that might account for diversity, e.g. try a contrastive search approach with a grid of parameters for k and alpha.
Comments, suggestions and typos: line 594: lemmata/lemmas instead of lemamas?
line 713: REF APPENDIX not properly formatted.
line 933: to Germany instead of to the Germany?
DistilBERT instead of distilBERT?
Overall assessment scores: 3.5