Package: nhssynth
Explore the docs »NHS SynthAboutThis repository currently consists of a Python package alongside research and investigative materials covering the effectiveness of the package and synthetic data more generally when applied to NHS use cases. See the internalproject descriptionfor more information.Getting StartedProject StructureThe main package and codebase is found insrc/nhssynth(seeUsagebelow for more information)Accompanying materials are available in thedocsfolder:The components used to create the GitHub Pagesdocumentation siteAreportsummarising the previous iteration of this projectAmodel cardproviding more information about the VAE with Differential PrivacyNumerous exemplar configurations are found inconfigEmptydataandexperimentsfolders are provided; these are the default locations for inputs and outputs when running the project using the providedCLImodulePre-processing notebooks for specific datasets used to assess the approach and other non-core code can be found inauxiliaryInstallationFor general usage, we recommend installing the package viapip install nhssynthin a supported python version environment. You can thenimportthe package'smodulesand use them in your projects, or interact with the package directly via theCLI, which is accessed using thenhssynthcommand (seeUsagefor more information).Secure ModeNote that in order to train a generator insecure mode(see thedocumentationfor details) you will need to install the PyTorch extension packagecsprngseparately. Currently this package's dependencies are not compatible with recent versions of PyTorch (the author's plan on rectifying this - watch this space), so you will need to install it manually; for this we recommend following the instructions below:gitclonegit@github.com:pytorch/csprng.gitcdcsprng
gitbranchrelease"v0.2.2-rc1"gitcheckoutrelease
pythonsetup.pyinstallAdvanced InstallationIf you intend on contributing or working with the codebase directly, or if you want to reproduce the results of this project, follow the steps below:Clone the repoEnsure one of the required versions of Python is installedInstallpoetryand either:Skip to step four (and havepoetrycontrol the installation's virtual environment in theirproprietary way)Changepoetry's configuration to manage your own virtual environments:poetryconfigvirtualenvs.createfalsepoetryconfigvirtualenvs.in-projectfalseYou can now instantiate a virtual environment in the usual way (e.g. viapython -m venv nhssynth) and activate it viasource nhssynth/bin/activatebefore moving to the next stepInstall the project dependencies withpoetry install(add optional flags:--with devwhen developing andtestingthe package,--with auxto work with theauxiliary notebooks,--with docsto work with thedocumentation)You can then interact with the package in one of two ways:Via theCLImodule, which is accessed using thenhssynthcommand, e.g.poetryrunnhssynth...Note that you can omit thepoetry runpart and just typenhssynthif you followed the optional steps above to manage and activate your own virtual environment, or if you have executedpoetry shellbeforehand.Through directly importing parts of the package to use in an existing project (from nhssynth.modules... import ...).UsageCLIThis package comprises a set of modules that can be run using theCLIindividually, as part of a pipeline, or via a configuration file. These options are available via the aforementioned(poetry run) nhssynthcommand:nhssynth <module name> --<args>
nhssynth pipeline --<args>
nhssynth config -c <name> --<overrides>To see the modules that are available and their corresponding arguments, runnhssynth --helpandnhssynth <module name> --helprespectively.Configuration files can be used to run the pipeline or a chosen set of modules. They should be placed in theconfigfolder and their layout should match that of the examples provided. They can be run as in the latter case above by providing their filename (without the.yamlextension). You can also override any of the arguments provided in the configuration file by passing them as arguments in the command line.To ensure reproducibility, you should always specify a--seedvalue and provide the--save-configflag to dump the exact configuration specified / inferred for the run (missing options will be populated in the outputted config, so it may be larger than one you would specify yourself). This configuration file can then be used in the future to reproduce the exact same run or shared with others to run the same configuration on their dataset, etc.Python APIAlternatively, you may want to import parts of the package into your own project or notebook. There is a minimum working example of thisin the auxiliary folder. You can learn more about the API and structure of the package and its modules in the docs to reuse components as you see fit.Package StructureThe figure below shows the structure and workflow of the package and its modules.View a visualisation of the codebasehere!RoadmapSee theopen issuesfor a list of proposed features (and known bugs). Ourmilestonesrepresent longer term goals for the project.ContributingContributions are welcome! We encourage you to first raise an issue with your proposed contribution to enable discussion with the maintainers. After that, please follow these steps:Fork the projectCreate your branch (git checkout -b <yourusername>/<featurename>)Commit your changes (git commit -m 'Add some amazing feature')Push to the branch (git push origin <yourusername>/<featurename>)Open a PR and we will try to get it merged!SeeCONTRIBUTING.mdfor detailed guidance.Thanks to everyone that has contributed so far!This codebase builds on previous NHSX Analytics Unit PhD internships contextualising and investigating the potential use of Variational Auto Encoders (VAEs) for synthetic data generation. These were undertaken by Dominic Danks and David Brind.LicenseDistributed under the MIT License.SeeLICENSEfor more information.ContactThis project is under active development by@HarrisonWilde. For feature requests and bugs, pleaseraise an issue; for security concerns, please open adraft security advisory. Alternatively, contactNHS England TDAU.To find out more about theAnalytics Unitvisit ourproject websiteor get in touch atengland.tdau@nhs.net.