Introduction to IBM Federated Learning: A Collaborative Approach to Train ML Models on Private Data | by Khuyen Tran | Towards Data Science

IBM Research has just released IBM Federated Learning on Github. If you have been looking for a way to boost your model training with the data collected from multiple resources while keeping the data private, this framework is for you. This article will explain why you need Federated learning and how to start using the framework.
When training a machine learning model, we often know about how your data looks, the distribution, the statistics of the data.
But in the real world, we might use the data from different sources without the privilege of knowing how the data is like. Why? Because data security is important in different enterprises.
For example, an aviation alliance wants to model how the recent events with COVID-19 will impact airline delays. Within the aviation alliance, we have several different airlines with different data. Since these airlines are competitors, the raw data cannot be shared across each airline. But the aviation alliance needs to use the combination of data to train the model!
There could be other non-competitive reasons are such as:
For some enterprises, moving data into one location can be impractical or expensive, particularly if the data are generated at a high velocity or the amount of data is large.
Even in the same company, customer databases might be in different countries and data cannot be taken out of the country.
Financial firms cannot share transaction data and medical institutions cannot share patient data due to regulation and secrecy.
So what is the solution? That is when we need Federated learning.
Federated learning (FL) is an approach to train machine learning models that do not require sharing datasets with a central entity. In federated learning, a model is trained collaboratively among multiple parties. The parties could keep their training dataset to themselves but still be able to participate in a shared federated learning process.
The learning process between parties most commonly uses a single aggregator. An aggregator would communicate with the parties and integrate the results of the training process.
For example, the process for neural networks could be:
Parties run a local training process on their training data, share the weights of their model with the aggregator.
These weights would then aggregate the weight vectors of all parties using a fusion algorithm.
Then, the merged model is sent back to all parties for the next round of training.
A federated decision tree algorithm might grow the tree in the aggregator, and query (request) the parties for the count information based on the parties’ data sets.
The whole process of FL is designed to ensure that a robust machine learning model is created from the collections of data without sharing the raw data of each party!
While FL does not require the centralized management of training data, it also poses some new changes such as:
Dataset heterogeneity: differences in the distribution, different dataset sizes for each party, different attributes (i.e. there might be more rich customers in one airline than another airline). Thus, there are no common properties of the overall training data set such as preprocessing methods, fine-tuning algorithms, and evaluation of model performance.
Privacy: some federated learning scenarios require that different parties (or even the aggregator) are not able to derive insights about each other’s training data based on messages exchanged during the training process (e.g., weights)
Skills: FL requires different skills from machine learning to distributed systems to cryptography. It’s rare to find employees who have all these skill sets.
Ideally, we want to solve the above problems without the difficulty of deploying, avoiding time-consuming processes such as opening ports as much as possible.
Fortunately, IBM Federated Learning framework addresses all of the above challenges and plus the ease of integration of FL into productive machine learning workflows!
The best way to know how a framework is to play with it! We would need more than one party to test this out. But don’t worry, we can simulate the scenarios of multiple parties trying to train their data from different machines! How good will the training perform? Stay tuned and observe the magic.
The steps below could be found in the setup section of IBM Federated learning Github repo
Start with cloning this repo. Then go to the folder created.
Install all the requirements. I really highly recommend using Conda installation for this project.
conda create -n <env_name> python=3.6conda activate <env_name>
Install the IBM FL package by running:
pip install <IBM_federated_learning_whl_file>
This example explains how to run federated learning on CNNs implemented with Keras training on MNIST data.
We will need more than one party to see how FL works. We can use generate_data.py to generate sample data on any of the integrated datasets. For example, we could run:
python examples/generate_data.py -n 2 -d mnist -pp 200
This command would generate 2 parties with 200 data points each from the MNIST dataset.
To run IBM federated learning, you must have configuration files for the aggregator and for each party. Each config file consists of the information about each party. You can generate these config files using the generate_configs.py script.
For example, you could run:
python examples/generate_configs.py -m keras_classifier -n 2 -d mnist -p examples/data/mnist/random
This command would generate the configs for the keras_classifier model, assuming 2 parties. -p is the path of party data.
Now put yourself in the seat of each party and the aggregator. Remember, in real life, the parties and the aggregator will not see one another’s screen but they will coordinate in the training process.
To start the aggregator, open a terminal window running the IBM FL environment set up previously.
In the terminal run:
In the terminal run:
python -m ibmfl.aggregator.aggregator examples/configs/keras_classifier/config_agg.yml
where the path provided is the aggregator config file path.
2. Then in the terminal, type START , and press enter. This means the aggregator starts accepting connections from other parties.
To register new parties, open a new terminal window for each party, running the IBM FL environment set up previously.
In the terminal run:
In the terminal run:
python -m ibmfl.party.party examples/configs/keras_classifier/config_party0.yml
where the path provided is the path to the party config file.
2. In the terminal for each party, type START and press enter. This means the party start accepting connections from the aggregator
3. Then in the terminal for each party, type REGISTER and press enter. This means the parties join the FL project
To initiate federated training, type TRAIN in your aggregator terminal, and press enter.
To synchronize model among parties, type SYNC
To save the model, type SAVE in your aggregator terminal, and press enter.
Voila! Now models for each party are saved to your folder. In the real scenario, each party would just see one model file for their party.
Now we have the model that is ready to be used for prediction!
Remember to type STOP in both the aggregator terminal and parties’ terminal to end the experiment process. Congratulation! Your party has just finished the training process without revealing your data.
Imagine you are party 0, to use the model to predict the test data of your party, all you need to do is to load the model and predict the test data of your party.
Awesome! Your party gets a f1-score of .81. You have not just secured your data but also get a robust machine learning model!
I will use the screenshot from IBM Data Science and AI Session 4 to compare the performance between the two
As you can see, all metrics including accuracy, f1, precision, recall of local model performance at party 1 are greater than the local model performance! By combining the data from other sources, you get a more robust model to predict your party’s data.
Congratulation! You have just learned how to train machine learning on the combination of datasets without sharing the data! Best of all, setting up training does not require advanced skills.
All you need is to add to your project file is a config that specifies the relevant information such as connection, data, model. Then use some simple command lines on the terminal to set up and train all parties at the same time.
The Github repository, as well as the literature for the framework, could be found here. You could also use IBM Federated Learning Slack channel to contact directly to the development team of the framework.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: