Package: dplab
DPLab: Benchmarking Differential Privacy Aggregation OperationsThis repo targets to provide a unified interface to access and evaluate the same aggregation functionalities in different open-source differential privacy (DP) libraries. With a simple CLI, one can choose the library, the aggregation function, and many other experimental parameters and apply the specified DP measurement to data stored in a.csvfile. The repo also provides both synthetic and real-world example datasets for evaluation purposes. Evaluation results are stored in a.jsonfile and metrics are provided for repeated experiments. The repo also provides a CLI tool to generate configuration groups for larger-scale comparison experiments.1-Min TutorialGet hands-on in 1 minute withour tutorial notebook.Currently supported aggregation operations:COUNTSUMMEANVARMEDIANQUANTILECurrently supported libraries:diffprivlib 0.5.2 [Homepage] [Example Usage]python-dp 1.1.1 [Homepage] [Example Usage]opendp 0.6.1 [Homepage] [Example Usage]tmlt.analytics 0.4.1 [Homepage] [Example Usage]chorus 0.1.3 [Homepage] [Example Usage]InstallationTo install dplab, one canuse the package on pypipipinstalldplabOrwith source code: clone the repo, switch the working directory, and install the dependenciesgitclonegit@github.com:camelop/dp_lab.gitcddp-lab
pipinstall-e.To usetmltexportPYSPARK_PYTHON=/usr/bin/python3
sudoaptinstallopenjdk-8-jre-headless
pip3install-ihttps://d3p0voevd56kj6.cloudfront.netpython-flint
pip3installtmlt.analyticsTo usechorus, please make sure you have Java runtime installed.
(If you have already installed tmlt, it should be fine.)How to run dp libraries in the benchmarkRun a specific library with the CLIdplab_run<library><operation><input_file><output_file><otheroptions>For example:dplab_runpydpsumdata/1.csvdata/1.json-f-r1000Other options include:mode: Evaluation mode, one can choose from "plain" (no timing/mem measurement), "internal" (internal measurement), or "external" (external tracking).epsilon: DP parameter, default is set to1.quant: Quantile value for QUANTILE operation, a float number between 0 and 1.lb: The optional value lower bound estimation used when applying certain differential privacy aggregations.ub: The optional value upper bound estimation used when applying certain differential privacy aggregations.repeat: How many time should the evaluation repeat.force: Force to overwrite the output file.debug: Include debugging information in the output file.python_command: Python command used to run the script in the external mode.external_sample_interval: timing/mem consumption sample interval in the external mode.For more information, please checkthe main entry file.Generating synthetic data# Make sure you are in the root directory of the repo# Data will be generated in the ./data/ directory# The procedure will generate about 28GB of data# To avoid the risk of running out of disk space, you can comment out the performance test lines (Line26-27) in SYN_TARGETS defined in the scriptpython3scripts/gen_data.pyHow to run experiments in the benchmarkGenerate the experiment commands, this will generate an./exp.db.jsonfile under the working directory (you can also use--locationto specify a different place).dplab_expplan--repeat100--group_num100Queue the experiments for executiondplab_explaunch--debugThe command updates the results toexp.db.json.One can potentially view the results viapython3scripts/view_exp_db.py