Package: bitfount
Federated learning and data analytics that just worksTable of ContentsUsing the Docker imagesRunning the Python codeInstallationGetting started (Tutorials)Federated training scriptsBasic Local UsageLicenseUsing the Docker imagesThere are two docker images, one for running a Pod (ghcr.io/bitfount/pod:stable),
and another for running a modelling task (ghcr.io/bitfount/modeller:stable).Both of the images require aconfig.yamlfile to be provided to them,
by default they will try to load it from/mount/config/config.yamlinside the docker container.
You can provide this file easily by mounting/binding a volume to the container,
how you do this may vary depending on your platform/environment (Docker/docker-compose/ECS),
if you have any problems doing this then feel free to reach out to us.Alternative you could copy a config file into a stopped container usingdocker cp.If you're using a CSV data source then you'll also need to mount your data to the container,
this will need to be mounted at the path specified in your config, for simplicity it's easiest
put your config and your CSV in the same directory and then mount it to the container.Once your container is running you will need to check the logs and complete the login step,
allowing your container to authenticate with Bitfount.
The process is the same as when running locally (e.g. the tutorials),
except that we can't open the login page automatically for you.Running the Python codeInstallationWhere to get itBinary installers for the latest released version are available at thePython
Package Index (PyPI).pip install bitfountFor DICOM support, you will need to install the DICOM extras:pip install 'bitfount[dicom]'If you want to use differential privacy (DP), you will need to install the DP extras:pip install 'bitfount[dp]'Ensure you are using python 3.8 or 3.9. The DP extra is not supported on 3.10.If you are planning on using thebitfountpackage with Jupyter Notebooks, we recommend you install the splinter packagebitfount[tutorials]which will make sure you are running compatible jupyter dependencies.pip install 'bitfount[tutorials]'Installation from sourcesTo installbitfountfrom source you need to create a python virtual environment.In thebitfountdirectory (same one where you found this file after cloning the git repo), execute:pip install -r requirements/requirements.inThese requirements are set to permissive ranges but are not guaranteed to work for all releases, especially the latest versions. For a pinned version of these requirements which are guaranteed to work, run the following command instead:#!/bin/bashPYTHON_VERSION=$(python-c"import platform; print(''.join(platform.python_version_tuple()[:2]))")pipinstall-rrequirements/${PYTHON_VERSION}/requirements.txtTo be able to use differential privacy (DP), you will need to additionally install the DP requirements. Please note that this is only compatible with Python version 3.8 and 3.9. Also, it is restricted to non-arm architectures:#!/bin/bashPYTHON_VERSION=$(python-c"import platform; print(''.join(platform.python_version_tuple()[:2]))")PLATFORM_PROCESSOR=$(python-c"import platform; print(platform.processor())")if[[${PYTHON_VERSION}=="38"||${PYTHON_VERSION}=="39"]]&&[[${PLATFORM_PROCESSOR}!="arm"]];thenpipinstall-rrequirements/${PYTHON_VERSION}/differential_privacy/requirements-dp.txtfiFor MacOS you also need to installlibomp:brew install libompGetting started (Tutorials)In order to run the tutorials, you also need to install the tutorial requirements:#!/bin/bashPYTHON_VERSION=$(python-c"import platform; print(''.join(platform.python_version_tuple()[:2]))")pipinstall-rrequirements/${PYTHON_VERSION}/requirements-tutorial.txtTo get started using the Bitfount package in a federated setting, we recommend
that you start with our tutorials. Runjupyter notebookand open up the first
tutorial in the "Connecting Data & Creating Pods folder:running_a_pod.ipynbFederated training scriptsSome simple scripts have been provided to run a Pod or Modelling job from a config file.⚠️ If you are running from a source install (such as fromgit clone) you will
need to usepython -m scripts.<script_name>rather than usebitfount <script_name>directly.To run a pod:bitfount run_pod --path_to_config_yaml=<CONFIG_FILE>To run a modelling job:bitfount run_modeller --path_to_config_yaml=<CONFIG_FILE>Basic Local UsageAs well as providing the ability to use data in remote pods, this package also enables local ML training. Some example code for this purpose is given below.1. Import bitfountimportbitfountasbf2. Create DataSource and load datacensus_income=bf.CSVSource(path="https://bitfount-hosted-downloads.s3.eu-west-2.amazonaws.com/bitfount-tutorials/census_income.csv",ignore_cols=["fnlwgt"],)census_income.load_data()3. Create Schemaschema=bf.BitfountSchema(census_income,table_name="census_income",force_stypes={"census_income":{"categorical":["TARGET","workclass","marital-status","occupation","relationship","race","native-country","gender","education"]}})4. Transform Dataclean_data=bf.CleanDataTransformation()processor=bf.TransformationProcessor([clean_data],schema.get_table_schema("census_income"))census_income.data=processor.transform(census_income.data)schema.add_datasource_tables(census_income,table_name="census_income")5. Create DataStructurecensus_income_data_structure=bf.DataStructure(table="census_income",target="TARGET",)6. Create and Train Modelnn=bf.PyTorchTabularClassifier(datastructure=census_income_data_structure,schema=schema,epochs=2,batch_size=256,optimizer=bf.Optimizer("RAdam",{"lr":0.001}),)nn.fit(census_income)nn.serialize("demo_task_model.pt")7. Evaluatepreds,target=nn.evaluate()metrics=bf.MetricCollection.create_from_model(nn)results=metrics.compute(target,preds)print(results)8. Assert resultsimportnumpyasnpassertnn._validation_results[-1]["validation_loss"]isnotnp.nanassertresults["AUC"]>0.7LicenseThe license for this software is available in theLICENSEfile.
This can be found in the Github Repository, as well as inside the Docker image.