Package: domias
DOMIAS: Membership Inference Attacks against Synthetic Data through Overfitting DetectionInstallationThe library can be installed from PyPI using$pipinstalldomiasor from source, using$pipinstall.APIThe main API call isfromdomias.evaluatorimportevaluate_performanceevaluate_performanceexpects as input a generator which implements thedomias.models.generator.GeneratorInterfaceinterface, and an evaluation dataset.The supported arguments forevaluate_performanceare:generator: GeneratorInterface
      Generator with the `fit` and `generate` methods. The generator MUST not be fitted.
  dataset: int
      The evaluation dataset, used to derive the training and test datasets.
  mem_set_size: int
      The split for the training dataset out of `dataset`
  reference_set_size: int
      The split for the reference dataset out of `dataset`.
  training_epochs: int
      Training epochs
  synthetic_sizes: List[int]
      For how many synthetic samples to test the attacks.
  density_estimator: str, default = "prior"
      Which density to use. Available options:
          * prior
          * bnaf
          * kde
  seed: int
      Random seed
  device: PyTorch device
      CPU or CUDA
  shifted_column: Optional[int]
      Shift a column
  zero_quantile: float
      Threshold for shifting the column.
  reference_kept_p: float
      Held-out dataset parameterThe output consists of dictionary with a key for each of thesynthetic_sizesvalues.For eachsynthetic_sizesvalue, the dictionary contains the keys:MIA_performance: accuracy and AUCROC for each attackMIA_scores: output scores for each attackdata: the evaluation dataFor bothMIA_performanceandMIA_scores, the following attacks are evaluated:"ablated_eq1" (Eq.1 (KDE))"ablated_eq2" (DOMIAS (KDE))"LOGAN_D1""MC""gan_leaks""gan_leaks_cal""LOGAN_0""eq1" (Eq. 1 (BNAF))"domias"Sample usageExample for usingevaluate_performance:# third partyimportpandasaspdfromsdv.tabularimportTVAE# domias absolutefromdomias.evaluatorimportevaluate_performancefromdomias.models.generatorimportGeneratorInterfacedefget_generator(epochs:int=1000,seed:int=0,)->GeneratorInterface:classLocalGenerator(GeneratorInterface):def__init__(self)->None:self.model=TVAE(epochs=epochs)deffit(self,data:pd.DataFrame)->"LocalGenerator":self.model.fit(data)returnselfdefgenerate(self,count:int)->pd.DataFrame:returnself.model.sample(count)returnLocalGenerator()dataset=...# Load your dataset as numpy arraymem_set_size=1000reference_set_size=1000training_epochs=2000synthetic_sizes=[1000]density_estimator="prior"# prior, kde, bnafgenerator=get_generator(epochs=training_epochs,)perf=evaluate_performance(generator,dataset,mem_set_size,reference_set_size,training_epochs=training_epochs,synthetic_sizes=[100],density_estimator=density_estimator,)assert100inperfresults=perf[100]assert"MIA_performance"inresultsassert"MIA_scores"inresultsprint(results["MIA_performance"])ExperimentsExperiments main paperTo reproduce results for DOMIAS, baselines, and ablated models, runcdexperiments
python3domias_main.py--seed0--gan_methodTVAE--datasethousing--mem_set_size_list30501003005001000--reference_set_size_list10000--synthetic_sizes10000--training_epoch_list2000changing arguments mem_set_size_list, reference_set_size_list, synthetic_sizes, and training_epoch_list for specific experiments over ranges (Experiments 5.1 and 5.2, see Appendix A for details) and gan_method for generative model of interest.or equivalently, runcdexperiments&&bashrun_tabular.shExperiments no prior knowledge (Appendix D)If using prior knowledge (i.e., no reference dataset setting), add--density_estimatorpriorExperiment images (Appendix B.3)Note: The CelebA dataset must be available in theexperiments/datafolder.To run experiment with the CelebA dataset, first runcdexperiments&&python3celeba_gen.py--seed0--mem_set_size4000and thencdexperiments&&python3celeba_eval.py--seed0--mem_set_size4000TestsInstall the testing dependencies usingpipinstall.[testing]The tests can be executed usingpytest-vsxCitingTODO