Privacy-Preserving Deep Learning with Federated Learning | Towards Data Science

Apple has recently announced a 47% YOY increase of its iPhone revenue with 49.26 million smartphones sold in the second quarter of 2021 [1]. For AAPL holders, this is absolutely good news. But what does it means for us, Data Science and AI enthusiasts?
Well, let’s do some calculations here. On average, every human creates about 1.7 MB (Megabyte) of data per second [2] and people usually spends approximately 3 hours on their phone every day. This means 848.46 PB (Petabyte) of data generated from iPhones on a daily basis which is 200 times bigger than the amount of data generated on Facebook every day (around 4 PB). Remember that this is just the amount of data from iPhones alone.
Although the numbers are not precise, you will now get the idea of how big this data is and why it becomes a new source of big data these days. As you might see, this copious source of data exceeds the size of information any single organisation can hold. These data can be used to create personalised experiences and it can help improve user satisfaction for the services or applications. Therefore, big organisations around the world are trying to utilise this new resource to increase the added value of their existing products or to create new business models around them.
However, there are many challenges arising when using this type of information. Unlike traditional big data source which is centrally stored and managed by an organisation, data on mobile devices is often distributed at a large scale and owned by the users instead of the service providers. And because the users own it, they have the right to keep it private on their devices due to privacy concerns and this is, therefore, the biggest challenge for the service providers who want to utilise this data. On one hand, the business needs more data to serve their customers better but on the other hand, private data cannot be shared.
Dealing with private and sensitive information is not easy, and the easiest approach to this problem is to not touch it. You rather don’t add the new feature using this data into the service than breach the user privacy and this is absolutely a valid point. However, if your competitors can somehow build AI applications with this data, provide better and personalised services without violating any privacy restrictions, then they will gain competitive advantages while you’re losing it. The technology that enables this capability will be a disruptive technology.
Feel convinced now? let’s keep reading!
Okay, so now you want to build an AI model on top of this gold mine of data without getting them out of users’ devices. When you think about data privacy and the related protections, encryption is one of the most popular methods in which data can be encrypted with user’s private key and then sent to the server. Nevertheless, encrypting also means you can no longer access the information since you don’t have the private keys from users. A rock is just a rock if you cannot get the gold inside.
The solution is indeed simpler than what you might think of. Instead of loading all the data to your server for model training, you can ship the model training to users’ devices where the data belongs and where your users have complete control of the data. This is the key idea of Federated Learning.
Federated Learning is a new paradigm of AI system architecture that essentially trains the model on client devices so that no private data will be sent out. There are generally four steps involved in the training process of Federated Learning:
Step 1: start training the model on all devices participating in the Federated Learning using their local data.
Step 2: devices send their locally trained models to the server. Note that it’s the model that is being uploaded instead of the private data. No private data will leave the devices.
Step 3: server combines all models received from devices by using an aggregation algorithm. There are many aggregation algorithms developed by researchers but the most popular algorithm is Federated Averaging published by Google in 2017.
Step 4: server distributes the aggregated model to all devices. The aggregated model can then be used to serve the AI features or be ready for the next training round if needed.
Learning from everyone without knowing about anyone
Aggregation algorithms play an important role in Federated Learning as it is responsible for combining the knowledge from all devices without knowing about users’ private data. In this article, I will only explain Federated Averaging (FedAvg) as it is a simple but effective algorithm that is being used in production for a Google’s product (although I am working on a new algorithm that can potentially beat the performance of FedAvg).
Federated Averaging algorithm produces the aggregated model by taking the weighted average of all model updates. To help you understand this better, I will provide an oversimplified example ignoring the “weighted” part of the algorithm. Given a system with three devices, here is how their models are aggregated.
- Device A sends model A with value 0.6 to the sever- Device B sends model B with value 0.1 to the sever- Device C sends model C with value 0.2 to the sever- Server takes the average of these three values: (0.6 + 0.1 + 0.2)/3 = 0.3- Server sends aggregated model with value 0.3 to A, B and C
And that’s it! The algorithm is simple but putting them all together, this idea has been proved to be “productionisable” by Google when they use it for the next word prediction feature of their Google Keyboard product on billions of Android devices. Their Natural Language Processing (NLP) model is super accurate since it learns from billions of users from multiple languages and cultures. However, the advantage here is that no typing data is collected and user privacy constraint is respected. This is the competitive advantage you might want for your data product.
You can check out the introduction from Google’s CEO, Sundar Pichai, about Federated Learning in this video:
Federated Learning is an emerging technology being adopted, researched and developed by many organisations around the world because of its enormous potentials. One can use Federated Learning to build a super-powerful diagnostic AI model for hospitals while reserving the privacy of the patients. One can also train self-driving cars using the driving behaviours of real drivers on the street. Personalised recommendation systems based on personal data can also be built to recommend products or services matching the customer’s needs accurately (most of the recommendation systems these days are product-centric, not customer-centric).
In terms of the maturity of this technology, Google has been using this in their products for a while now. Besides, there are many Federated Learning frameworks backed by big companies so that you don’t start from scratch when adopting this technology. Two frameworks that I have tried myself are OpenFL by Intel and TensorFlow Federated by Google. These frameworks provide easy-to-use programming interfaces so that your Data Scientists can build and deploy their first Federated Learning application with just a small piece of code.
Every great opportunity comes with great challenges.
Nevertheless, the biggest challenges, as usual, do not come from the technology itself but instead arise from the overall design of the product implementing the technology solution.
When Google and Apple apply a new technology in Android and iOS, you’re opt-in by default as a user. They never ask you something like: “Will you give us access to your private and sensitive data to help improve your experience? We promise to not send your data out”. The minute you see this popup, I guess you will press “No Thanks” immediately.
Since Google and Apple own the Operating System (OS), the users have no choice unless they stop using the device. However, as a company develops applications on top of the OS, you don’t have that luxury. Every time the app needs access to user’s data, it will still have to specifically ask for permission even though the data won’t be uploaded somewhere else. Imagine if you have to explain the “we only send the model out” part to your customers to get their permissions, you will realise that not only this is impossible but also the reputation of your organisation might be impacted as you sounds like a spy company unless you do it the way Facebook does (you know what I mean). Otherwise, your products or services must be valuable enough so that the users are willing to “compromise” their data even though your intention is to keep the data on their devices. When I use the word “compromise”, I am expressing the thinking of the user rather than the intention of the service provider. For example, everyone is making compromises when using Facebook since they are all aware that it is collecting their data but still, they keep using it.
While customer’s consent is a challenge for B2C, it is an easier problem to solve for B2B. Two or more companies can agree upon a deal of sharing data that is beneficial for all parties and, in fact, this approach is quite popular these days. Now with Federated Learning, they can even share the data without actually sending it out. For instance, an Internet Service Provider (ISP) can exchange the internet traffic data of its users with a bank to build some advanced AI applications bringing values to both sides.
The ISP can build a demand prediction model using its own data enriched by the transaction data from the bank. There might be a connection between the expenditure of a household with their internet usage. An example of this could be that a family have just visited a home entertainment retail store (those selling tablets, laptops and game consoles) and made the payment worth around $3000. That could also mean the family now have more smart devices at home and hence they will need more internet bandwidth. The prediction model can help the ISP offers better plans to their customers and be more initiative in their infrastructure planning.
On the other hand, the bank can build a better cash flow forecasting model for their institutional clients (retail stores for example) using transaction data enriched with internet traffic data provided by the ISP. There might be a correlation between the amount of traffic to a specific website and the next-week revenue of the company owning that website. For example, the ISP reports an increase in traffic to the address www.theretailstore.com this week, the bank can then combine this information with the historical transaction data of this retail store to predict the revenue of next week because maybe the customers are looking for information on the website before they purchase the products.
There are a lot more applications and ideas I can list out here and the potential is limitless. So whether it is B2B or B2C, you might want to think about the values you will get out of the data you can never access before because of the privacy concerns and investing in this technology in order to get competitive advantages against your competitors. Every great opportunity comes with great challenges and those who can manage the risks arising from those challenges will get the opportunity and disrupt the market.
Apple sales miss expectations, Tim Cook says supply issues cost company $6 billionHow Much Data Is Created Every Day in 2021?Federated Learning: Collaborative Machine Learning without Centralized Training DataFederated Learning with GoogleIntel® Open Federated LearningTensorFlow Federated: Machine Learning on Decentralized DataCommunication-Efficient Learning of Deep Networks from Decentralized Data
Apple sales miss expectations, Tim Cook says supply issues cost company $6 billion
How Much Data Is Created Every Day in 2021?
Federated Learning: Collaborative Machine Learning without Centralized Training Data
Federated Learning with Google
Intel® Open Federated Learning
TensorFlow Federated: Machine Learning on Decentralized Data
Communication-Efficient Learning of Deep Networks from Decentralized Data