Struct-Bench: A Benchmark for Differentially Private Structured Text
  Generation
Authors: name: Shuaiqi Wang, name: Vikas Raunak, name: Arturs Backurs, name: Victor Reis, name: Pei Zhou, name: Sihao Chen, name: Longqi Yang, name: Zinan Lin, name: Sergey Yekhanin, name: Giulia Fanti

Differentially private (DP) synthetic data generation is a promising
technique for utilizing private datasets that otherwise cannot be exposed for
model training or other analytics. While much research literature has focused
on generating private unstructured text and image data, in enterprise settings,
structured data (e.g., tabular) is more common, often including natural
language fields or components. Existing synthetic data evaluation techniques
(e.g., FID) struggle to capture the structural properties and correlations of
such datasets. In this work, we propose Struct-Bench, a framework and benchmark
for evaluating synthetic datasets derived from structured datasets that contain
natural language data. The Struct-Bench framework requires users to provide a
representation of their dataset structure as a Context-Free Grammar (CFG). Our
benchmark comprises 5 real-world and 2 synthetically generated datasets, each
annotated with CFGs. We show that these datasets demonstrably present a great
challenge even for state-of-the-art DP synthetic data generation methods.
Struct-Bench also includes reference implementations of different metrics and a
leaderboard, thereby providing researchers a standardized evaluation platform
to benchmark and investigate privacy-preserving synthetic data generation
methods. Further, we also present a case study showing how to use Struct-Bench
to improve the synthetic data quality of Private Evolution (PE) on structured
data. The benchmark and the leaderboard have been publicly made available at
https://struct-bench.github.io.