End-to-End on-device Federated Learning: A case study
Authors: Hongyi Zhang
Jan Bosch
Helena Holmström Olsson
Venue:  (2021)
Keywords: Federated Learning
Machine Learning
End-to-End Learning
Artificial Intelligence

Abstract: With the development of computation capability in devices, companies are eager to utilize ML/DL methods to improve their service quality. However, with traditional Machine Learning approaches, companies need to build up a powerful data center to collect data and perform centralized model training, which turns out to be expensive and inefficient. Federated Learning has been introduced to solve this challenge. Because of its characteristics such as model-only exchange and parallel training, the technique can not only preserve user data privacy but also accelerate model training speed. In this paper, we introduce an approach to end-to-end on-device Machine Learning by utilizing Federated Learning. We validate our approach with an important industrial use case, the wheel steering angle prediction in the field of autonomous driving. Our results show that Federated Learning can significantly improve the quality of local edge models and reach the same accuracy level as compared to the traditional centralized Machine Learning approach without its negative effects. Furthermore, Federated Learning can accelerate model training speed and reduce the communication overhead, which proves that this approach has great strength when deploying ML/DL components to real-world embedded systems.

Under review as a conference paper at ICLR 2021
END-TO-END ON-DEVICE FEDERATED LEARNING: A CASE STUDY
Anonymous authors Paper under double-blind review
ABSTRACT
With the development of computation capability in devices, companies are eager utilize ML/DL methods to improve their service quality. However, with ditional Machine Learning approaches, companies need to build up powerful data center to collect data and perform centralized model training, which turns out be expensive and inefficient. Federated Learning has been introduced solve this challenge. Because of its characteristics such as model-only exchange not only preserve user data privacy but
and parallel training, the technique also accelerate model training speed. In this paper, we introduce an approach to end-to-end on-device Machine Learning by utilizing Federated Learning. We val- idate our approach with an important industrial use case, the wheel steering angle prediction in the field of autonomous driving. Our results show that Federated Learning can significantly improve the quality of local edge models and reach the same accuracy level as compared the traditional centralized Machine Learn- ing approach without its negative effects. Furthermore, Federated Learning can accelerate model training speed and reduce the communication overhead, which proves that this approach has great strength when deploying ML/DL components to real-world embedded systems.
INTRODUCTION
With the development of computation capability in devices, Machine Learning and Deep Learning arouse great interests by companies who are eager to utilize ML/DL methods to improve their service quality. However, with the explosive growth of data generated on edge devices, the traditional centralized Machine Learning approaches have shown its weakness, such data communication overhead, model compatibility, training efficiency, etc. (L'heureux al.) 2017a)) Figure illlustraee traditional Machine Learning approach with the centralized learning framework.
Edge Device
Deployment
Training
Data
Model Trainer
Edge Device
Collection
Model Deployment
Model
Figure 1: Traditional Centralized Learning System
The diagram contains four stages: 1) data collection from multiple distributed edge devices 2) model training in central server 3) model validation based existing testing data 4) model deployment

---

Under review as a conference paper at ICLR 2021
to edge devices. However, the data collected from edge devices need to be transmitted to a central server and perform model training on that enormous data set, which turns out to be inefficient and expensive. In order to solve these challenges, Federated Learning has been introduced as efficient approach which can distribute learning tasks to the edge devices and avoid massive data transmis- sion. Furthermore, due to the characteristies of Federated Learning, on-device training becomes possible and the local model quality can be continuously improved.
Although the concept of Federated Learning has significant benefits and potential in Al engineering fields, it is hard for industries and companies to build a reliable and applicable on-device Federated Learning system. Some previous research identified the challenges of deploying AI/L components into a real- real-world industrial context. As defined in "Engineering Al Systems: A Research Agenda" (Bosch al. 2020), AI engineering refers to AI/MI-drriven software development and deployment in production contexts. We found that the transition from prototype to the production-quality de- ployment of ML models proves to be challenging for many companies (I. (L'heureux al. (017b)
(wakatare et al. 2019).
The contribution of this paper is threefold. First, we utilize Federated Learning, a distributed ma- chine learning technique, and validate it on important industrial use case, steering wheel pre- diction in the field of autonomous driving, which is also a classic end-to-end learning problem. Second, describe end-to-end on-device Federated Learning approach to efficiently train Ma- chine Learning models in a distributed context. Third, we empirically evaluate our approach on the real-world autonomous driving data sets. Based our results, we demonstrate the strength of Federated Learning compared to traditional centralized learning methods.
The remainder of this paper is structured as follows. Section 2 we introduce the background of this study. Section 3 details our research method, including the simulation testbed, the utilized machine learning method and the evaluation metrics. Section 4 presents the end-to-end Federated Learn ing approach utilized in this paper. Sections evvaluates proposed learning approach to empirical data sets. Section o outlines the discussion on our observed results. Finally, Section 7 presents conclusions and future work.
2 BACKGROUND
The first Federated Learning framework was proposed by Google in 2016 Konečný et al. (2016), The major objective of Federated Learning is to learn a global statistical model from numerous edge devices. Particularly, the problem to minimize the following finite-sum objective function ||
min [ww, where f(w) = Σ Xxs((w)
(1)
Here, represents model parameters, is the total number of edge devices, and fi(w) is the local objective function which is defined by high dimensional tensor w of the ich device. X (A; 2 0 and Σ; >; 1) gives the impact of izh remote device and is defined by users. This formula is also applied throughout this research.
With the development of the concept of cloud computing and decentralized data storage, there has been increasing interest in how to utilize this technique to improve Machine Learning procedure. There are two classic applications which were realized by Hard et al. (2018) and Ramaswamy et al. (019). Authors applied Federated Learning techniques on the Google Keyboard platform to im- prove virtual keyboard search suggestion quality and emoji prediction. Their results show feasi- bility and benefit of applying federated learning to train models while preventing to transfer user's data. However, authors previous research didn't discuss the impact of model training time and the communication cost when deploying and training models edge devices. Furthermore, due to the system environment and troubles encountered when deploying Federated Learning into differ- ent cases, propose end-to-end approach and validate the on-device Federated Learning into completely different industrial scenario, the steering wheel angle prediction.
With the inspiration of the work by Bojarski al. (2016), designed and developed a deep con- volutional neural network to directly predict the steering wheel angle and control the steer based on the prediction. The training data is collected from single images sampled from video and the
2

---

Under review as a conference paper at ICLR 2021
ground truth is recorded directly from real-time human behavior. In order to improve the model prediction performance, a two-stream model was first proposed in Simonyan & Zisserman (2014) and applied in Fernandez (2018) due to its robustness and lower traiing cost compared with other networks such as 3D-CNN (Du et al. 2019). RNN (Fraqi et al. 2017) and L.TT (Valiente et al.) 2019). However, the previous research for this case is mainly focusing single vehicle. In this paper, we will apply Federated Learning to accelerate model training speed and improve the model quality by forming global knowledge of all participating edge vehicles.
training model in
METHOD
In this research, the empirical method and learning procedure described in Zhang & Tsai (2003) was applied to make a quantitative measurement and comparison between Federated Learning and traditional centralized learning methods. In the following sections, present the mathematical no- tations used in this paper, our testbed, data traces and the convolutional neural network architecture utilized for solving the problem of steering wheel angle prediction.
3.1 MATHEMATICAL NOTATIONS
We first introduce the mathematical notations that will be used in the rest of the paper:
A.
An image frame matrix at time
O. f(Ar, At-1)
An optical-flor fiical-floo matrix time
0,
Steering wheel angle at time
3.2 DATA TRACES AND TESTBED
The datasets used in this paper SullyChen collection of labeled car driving data sets, which is available on Github (hullyChen 2018). In this collection, there are two datasets (Dataset 2017 and Dataset 2018) which record different driving information on different routes.
Dataset 2017 contains approximately 45,500 images, 2.2 GB. The dataset records trajectory of ap- proximately 4km around the Rolling Hills LA, USA in 2017. 2017 dataset is used for pretraining the model. (The model will be used to initialize edge models before Federated Learning)
Dataset 2018 contains approximately 63,000 images, 3.1 GB. This dataset records trajectory of approximately 6km along the Palos Verdes in LA. 2018 dataset is used for end-to-end Federated Learning and model validation. In order to provide fruitful evaluation, we conducted experiment on 8, 16, 32 and 64 edge vehicles. The data were divided into corresponding number of parts and distributed to edge vehicles. Besides, in each edge vehicle, 70% of local dataset were training set while 30% were acted as the testing
In each edge vehicle, the first 70% data are regarded as the previously recorded driving information while the rest 30% are future information. The models were continuously trained based the recorded information and perform prediction and validation on the steering wheel angle information by using future driving data.
Table I provides the hardware information for all of the servers. In order to simulate aggregation and edge functions, one server is adopted as the aggregation server while the rest acted as edge vehicles.
3.3 MACHINE LEARNING METHOD
A two-stream deep Convolutional Neural Network (CNN) (Simonyan & Zisserman 2014) (Fernan- dez 2018) is utilized to perform angle prediction. Figure 2 gives the detailed information about the architecture. In our implementation. each stream has two convolutional layers and max-poot max-pooling layer. After concatenating, there are two tully-connected layers which are activated by ReLU func- tion.
3

---

Under review as a conference paper at ICLR 2021
Table 1: Hardware setup for testhed servers
CPU
Intel(R) Xeon(R) Gold 6226R
Cores
8
Frequency
2.90 GHz
Memory
32 GB
OS
Linux 4. 4.15.0-100 -00ggerree
GPU
Nvidia Tesla T4 GPU (Only in edge vehicles)
Frame Stream
Input
ELU0
1x stride=2
24 3x3 stride=2
Maxpool 4x4 stride=2
ReLuO
ReLun
Output Angle
Optical Flow Stream Input
concatO
256
10
ELUD
12 stride=2
24 stride=2
Maxpool 4x4 stride=2
Figure 2: Model description
The model contains two different neural branches which consume spatial information and temporal information as the inputs of two streams and then output the predicted steering angle. For the first stream the model consumes 3 frames of RGB images, which can be denoted as {A,-2. A,- Ar). The second stream is the two-frame optical flow calculated by two consecutive frames O,-1 [[[A,2, 1)-1)) and O, [({AA-1,A:))
Optical flow is a common temporal representation in video streams, which captures the motion changes between two frames (Horn Schunck 198T). The method of calculating optical flow ap- plied in this paper is based on Gunnar Farneback's 's algorithm implemented in OpenCV (Farneback) 2003]. Figure demonstrate an example optical flow matrix produced by two consecutive image frame.
The process of training an local CNN network is to find the best model parameters which cause the minimum difference between the predicted angle and the ground truth steering angle. Therefore, in this case, we choose mean square error as the local model training loss function:
Loss
1 N
>
(0. 0.)
Here, N represents the batch size while 0, and 0, represent the ground truth and the predicted steering wheel angle value at time t.
During the process of model training in each edge vehicles, all the image frames will be firstly normalized to 1]. The batch size is 16 while the learning rate is set to le 5. The optimizer utilized here is Adam (Kingma Ba 2014), with parameters 31 0.6, 3r 0.99 and le 8.
3.4 EVALUATION METRICS AND BASELINE MODEL
In order to provide fruitful results and evaluation. we selected three metrics and two baseline models. The three metrics includes angle prediction performance, model training time and bandwidth cost:
4

---

Under review as a conference paper at ICLR 2021
(a) A:-1
(b)
(c) O, (({A,-1,
Figure 3: Example of the optical flow
Angle prediction performance: We root mean square (RMSE), common met- ric, to measure the difference between prediction results and ground truth. The metrics can provide good estimation about the quality of trained model in each edge vehicles.
Model training time: This metric is defined the time cost for training model at the edge vehicles. The result is the average of four edge vehicles during one training round. This metric demonstrates the speed of local edge devices updating their knowledge which is crucial and important for those systems which need to quickly evolve to adapt to the rapidly-changer environment. The metrics measured in all the vehicles by checking model deployment timestamp.
Bandwidth cost: This metric is defined as the total number of bytes transmitted during the whole training procedure. This metric demonstrate the total communication resources cost achieving an applicable CNN model.
The two baseline models includes model trained by applying traditional centralized learning ap- proach and the locally trained model without model sharing:
Traditional Centralized I. Learning model (ML.) This baseline model is trained under the traditional centralized learning approach. Before model training. all the data from edge vehicles are firstly collected to a single server. The hyper-parameter of this model training is the same as Federated Learning which is mentioned section [3.]] The performance can be then compared with the model trained by Federated Learning approach.
Locally trained model without model sharing (Local ML):
This baseline models are trained directly each edge vehicles. However, different from Federated Learning, there will be no model exchange during the training procedure. The prediction performance can be compared with Federated Learning model to see how Fed- erated Learning can outperform those independently trained local models.
END-TO-END FEDERATED LEARNING
In this section, we describe the algorithm and the approach applied in this paper. In order to perform on-device end-to-end learning based on the input image frames, images are firstly stored in an ex- ternal storage driver located on each edge vehicles. At the same time, the optical flow information calculated. When triggering the training threshold, image frames and optical flow frames are fed into a convolutional neural network. The output of the network is compared to the ground truth for that image frame, which is the recorded steering wheel angle. The weights of the CNN are adjusted using back propagation to enforce the model output as close as possible to the desired output. Figure iillustrate the diagram of the learning procedure in single edge vehicle.
5

---

Under review as a conference paper at ICLR 2021
o
Input Steering Wheel Angel
Error Back Propagation
Network Networkk predicted Angle
Image Frames
Optical Flow Frames
Exchange and Aggregate with other
Figure 4: Diagram of end-to-end on-device learning procedure in single vehicle
After finishing each training epoch, models in edge vehicles will also be updated to the aggregation server and form a global knowledge among other cars (Figure 5). The aggregation applied this paper is FedAvg (Li et al. [2019), which is commonly used Federated Learning algorithm in most of the research. The steps FedAvg algorithm is listed below:
Step
Edge vehicles locally compute the model; After finishing each five local training epoch, they send updated model results to the aggregation server.
Step 2:
The central server performs aggregation by averaging all updated models to form a global knowledge of all local models.
Step 3:
The aggregation server sends back the aggregated result to each edge vehicles.
Step 4:
Edge vehicles replace the local model and performs further local training by using the global deployed model.
Aggregation
Server
Model Update
Model Update
Model Update
Model Update
Local Training
Local Training
Local Training
Local Training
Figure 5: Process of Federated Learning
EVALUATION
In this section, we present the experiment results of the presented end-to-end on-device Federated Learning approach the formance three aspects (The metrics are defined in section 3.4p (1) Angle prediction performance (2) Model Training Time (3) Bandwidth cost.
of steering wheel angle prediction. We evaluate the system per-
Figure (illustrrte the angle prediction performance between the model trained by Federated Learn- ing (FL.) and the locally trained model without any model exchange (Local MI.). The results demon-

---

Under review as a conference paper at ICLR 2021
strate that the performance of traditional centralized trained model behaves similar to Federated Learning model. Besides, compared with independently trained model, Federated Learning can provide better prediction which is much closer to the ground truth.
We show detailed results with 4 vehicles participated in Federated Learning, which provides clear view of prediction performance in each edge vehicle. The results illustrate that in vehicle and 4, model of Federated Learning outperform other baseline models. In vehicle and 3, model of Federated Learning only perform about 1 worse than the traditional centralized learning model. Based Learning model can provide more accurate prediction than local independently trained model and the behaviour of Federated Learning model can reach the same accuracy level compared with centralized learning model.
Numeric results are provided in Table 2
results,
summarize that Federated
Stering wiree tage (Drggne)
wwg.e (Drgroc)
-50
-15
-10
-100
1000
15000
2000
3000
1000
1900 Timestamp
2000
2500
3000
Trrestirrp P
Vehicle 1
(b) Vchicle
80-
Steetig Whenl logge (begre))
40-
>0-
40
-09
300
3000
(c) Vehicle 3
(d) Vehicle
Figure 6: The comparison of angle prediction performance on four local vehicle test set with Feder- ated Learning and two baseline models
Vehicle
Vehicle
Vehicle
Vehicle
Overall
FL
3.154
8,875
17.209
5.581
10.242
ML
5.371
7.914
16.215
7.258
10.099
Local ML
4.017
14.775
25.670
7.313
15.419
Table 2: Steering wheel angle regression error (RMSE) test of each edge vehicle (4 vehicles in total)
Tlive the comparison of total training time and bytes transferred between Federated Learning and two bascline model. The total number of training epochs for all the models is 100 and the model training is accelerated by Nvidia Tesla T4 GPU. The results show that Federated Learning need slightly more training time than independently locally trained model due the model exchange time cost. However, the training time of Federated Learning is reduced about 75% and we save about 25% bandwidth compared with traditional centralized learning method.
In order to evaluate the impact of different number learning vehicles, we perform more experi- ments with 8, 16, 32, 64 vehicles participated. Table 4 gives the overall steering angle prediction

---

Under review as a conference paper at ICLR 2021
FL.
MI.
Local ML.
Total Training Time (sec)
511.6
2137.2
485.3
Total Bytes Transferred (GB)
1.56
2.02
Table 3: Training Time and Bandwidth cost with different model training methods (4 Vehicles in total)
emorr and total training time of Federated Learning model with different number of vehicles. The overall value provides an overview prediction performance among of the test datasets belongs to all vehicles. With the increasing number of edge vehicles, the model prediction performance the edge is further enhanced. Furthermore, total model training time is linearly decreased corre- sponding the increasing number of edge vehicles. Based with the participation of more edge vehicles and the larger size of the input datasets, the advantages of Federated Learning will become more obvious.
results,
summarize that
Number of Vehicles
4
16
32
64
Error (RMSE)
10.242
10.649
9.644
9.387
9.251
Total Training Time (sec)
511.6
264.3
125.1
65.3
31.7
Table 4: Overall steering wheel angle regression error (RMSE) and model training time of Federated Learning model with different number of vehicles participated
6 DISCUSSION
Based on our experiment results, end-to-end on-device Federated Learning approach has more ad- vantages compared with commonly used centralized learning approach. Federated Learning model can achieve same level of model prediction accuracy but decrease model training time and the band- width cost. Furthermore, if compared with independently local trained model, because of the model sharing mechanism, Federated Learning can form global knowledge the whole datasets which are belongs to different participated edge vehicles. The model quality are largely enhanced and can achieve much better results.
Due to those advantages, there variety of other meaningful use cases that end-to-end on-device Federated Learning can help. The technique reported this paper can not only be used for steer- ing angle prediction in self-driving vehicles but also other on-device applications, such camera sensors and motion detection, which requires continuously machine learning model training on the resource-constrained edges. Furthermore, because of the user data privacy and network bandwidth constraints, Federated Learning can be applied in those systems which need quickly-evolved model adapt their rapidly changing environment.
7 CONCLUSION
In this paper, describe an approach to end-to-end on-device Machine Learning by utilizing Feder- ated Learning. We validate our approach with the wheel steering angle prediction in the self-driving vehicles. Our results demonstrate the strength and advantage of the model trained under end-to- end Federated Learning approach. The model can achieve the same level of prediction accuracy compared with commonly used centralized learning method but reduces training time with 75% and bandwidth cost with 25 % in case. Note that if the number of participating devices is further increased, the reduction will be more obvious and the strength of Federated Learning will become stronger.
In the future we plan to validate our approach in more use cases. Also, we would like to explore more advanced neural network combined with Federated Learning method. Furthermore, we plan to find more suitable aggregation algorithms and protocols for our end-to-end on-device Federated Learning approach.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End end learning for self-driving cars. arXiv preprint arXiv: 1604.03366, 2016.
Jan Bosch, Ivica Crnkovic, and Helena Holmström Olsson. Engineering ai systems: A research agenda. arXiv preprint (rXi:22001.05522 2020.
Shuyang Du, Haoli Guo, and Andrew Simpson. Self-driving car steering angle prediction based on image recognition. arXiv preprini arXiv: 1912.:/9/2.004400 2019.
Hesham M Eraqi, Mohamed N Moustafa, and Jens Honer. End-to-end deep learning for steering au- tonomous vehicles considering temporal dependencies. arXiv preprint arXiv:1710.03800 2017.
Gunnar Farnbäcck. Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis, pp. 363-370. Springer, 2003.
Nelson Fernandez. Two-stream convolutional networks for end-to-end learning of self-driving cars. arXiv preprint arXiv: arXiv:/81 181::07811.... 2018
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 2018.
Berthold KP Horn and Brian G Schunck. Determining optical flow. Techniques and Applica- lions of Image Understanding, volume 281, pp. 319-331. International Society for Optics and Photonics, 1981.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:/610.02553 2016.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv: 1907.0218,, 2019.
Lucy Ellen Lwakatare, Aiswarya Raj. Jan Bosch, Helena Holmström Olsson, and Ivica Crnkovic. A taxonomy of software engineering challenges for machine learning systems: An empirical in- vestigation. In International Conference on Agile Sofware Development, pp. 227-243. Springer, Cham, 2019.
Alexandra L'heureux, Katarina Grolinger, Hany F Elyamany, and Miriam AM Capretz. Machine learning with big data: Challenges and approaches. IEEE Access, 5:7776-7797, 2017a.
Alexandra L'heureux, Katarina Grolinger, Hany F Elyamany, and Miriam AM Capretz. Machine learning with big data: Challenges and approaches. IEEE Access, 5:7776-7797, 2017b.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated learning for emoji prediction in mobile keyboard. arXiv preprini arXiv: 9906.04399, 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 668-576,15 2014.
SullyChen. Collection of labeled car driving datasets, 2018. URL. http:: com/
SullyChhe/drivvngg-- www.wiiinnddttttteee.e.ee
Rodolfo Valiente, Mahdi Zaman, Sedat Ozer, and Yaser Fallah. Controlling steering angle for cooperative self-driving vehicles utilizing and Istm-based deep networks. In 2019 IEEE In- telligent Vehicles Symposium (IV), pp. 2423-2428. IEEE, 2019.
Du Zhang and Jeffrey JP Tsai. Machine learning and software engineering. Software Quality Jour- nal, 11(2):87-119, 2003.


Reviews:
[{"review_id": "kt5JZTV0Z86", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The study evaluates federated learning (FL) in the context of steering wheel angle prediction, which is relevant for autonomous driving systems. Authors compare against two baselines a centrally-computed and locally-computed models and measure prediction error, training time and bandwidth cost. The work evaluates an existing approach and therefore its novelty and impact is limited. It does provide an interesting evaluation of FL for a relevant use case. Federated learning, as the authors indicate in the manuscript, is a promising approach for training ML applications while preserving user privacy, which is key to many industrial ML applications such as voice assistants and computer vision algorithms. For that reason, the impact of the paper is significant despite not being very original. The authors carry out a very simple study, but which seems sufficient to demonstrate that FL can have computational advantages, namely reduced training times and bandwidth costs. A challenging application of FL are ML applications that run on small devices that people carry around all the time, such as mobile phones and wearable devices.  In that scenario, there is the additional constraint that resources for training models on device are typically limited,  the smaller the device the more limited. An interesting extension of this study would be evaluate amount of computational resources used on the device as an additional evaluation metric. It would be great if the authors could add this metric to the present paper, but it could also be something for a followup publication, in other words, I do not think is needed for this paper to be published.   \n\n[Update after author's rebuttal]\nI do not see any reason to modify my rating. I also identified the self-citation , but it did not affect my rating or evaluation of the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PUTR6RM85_v", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "******************************************************************************\n\nThe paper is not properly anonymized. The intro refers to “Our previous research” and says “As we defined in Engineering AI Systems: A Research Agenda, (Bosch et al., 2020), … .” As such it violates the anonymity policy. \n******************************************************************************\n\nThis paper describes end-to-end implementation of Federated Learning (FL) on a use case of steering wheel prediction in autonomous driving. It provides empirical evaluation on real-world autonomous driving datasets and shows improved performance compared to centralized learning methods. \n\nPros:\nIs it interesting to see an implementation of FL on a real-world use-case. The paper also does well in comparing different factors such as training time and bandwidth cost for FL and centralized training.\n\nCons:\nThe paper doesn’t have enough technical depth to be accepted at ICLR and reads more like a report than a paper. It mainly describes the implementation of FL for a real-world application, which, although important, does not contribute to the field in terms of developing better algorithms or better understanding the current ones.  \n\nA large part of the experiment section describes the hardware features, network structure and training method in great details, which seems redundant or unnecessary for an ICLR submission. For example, section 4 reads “The weights of the CNN are adjusted using back propagation to enforce the model output as close as possible to the desired output.”, which is obvious to most readers.  \n\nThere are also some statements in paper that are not quite scientific or concrete. For example, the intro reads “due to the characteristics of Federated Learning, on-device training becomes possible.” This is not true as on-device training is not becoming possible due to FL, though FL certainly requires it. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Icsyeh_VTOk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies federated learning to  steering wheel prediction for autonomous driving. \"Federated learning\" in this draft mainly refers to an on-device distributed training algorithm where each edge device hosts its private data and performs local updates (model training) and send the updates back to a central server to aggregate. More specifically, this paper uses the most well-known algorithm in federated learning, FedAvg (McMahan et al. 2017). \n\nPros\n+ The application is real and seems important. \n+ Distributed/federated learning makes sense for this application. \n\nCons\n- The main contributions of the draft are not clear. It looks to me such empirical studies of a well-known algorithm on a specific application will better fit a more application-oriented or system-oriented venue, e.g., CVPR, SysML. \n- How are the hyperparameters tuned for centralized and federated setting? \n- What are the hardwares on edge devices/vehicles, and what are the hardwares in datacenter for centralized training? The draft mentioned Tesla T4 GPUs, but it seems not clear exactly how much computation power has been used. \n- Could the authors clarify \"companies need to build up a powerful data center to collect data and perform centralized model training, which turns out to be expensive and inefficient. Federated Learning has been introduced to solve this challenge.\"? As far as I know, the primary motivation for federated learning is privacy protection. Edge devices has far less computation power and big communication barrier, why would it solve \"this challenge\"?\n- The following sentences seem to against the anonymous rules? \"Our previous research shows the challenges of deploying AI/ML components into a real-world industrial context. As we defined in ”Engineering AI Systems: A Research Agenda” (Bosch et al., 2020), AI engineering refers to AI/ML-driven software development and deployment in production contexts. We found that the transition from prototype to the production-quality deployment of ML models proves to be challenging for many companies (L’heureux et al., 2017b) (Lwakatare et al., 2019).\"\n\nSome minor improvement:\nThe abbreviation “ML/DL” seems never introduced\nIt seems unnecessary to capitalized “Machine Learning”, “Federated Learning”. \nConsider cite the original FedAvg (McMahan et al. 2017) paper instead of (Li et al., 2019).\n\n====== post rebuttal ======\n\nI do not think the response addressed my concerns. I would strongly suggest authors reconsider the design choices where I raised questions. Note that these are not only clarification questions, but also fundamentals of machine learning and federated learning.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8pP3s7gp_yX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a case study that applies Federated Learning for steering angle prediction in self-driving cars. All methods used have been previously proposed in the literature.\n\nPros:\n+ A case study for an industrial use of federated learning (in autonomous driving application).\n+ Results do show that Federated Learning can give accuracy close to a centralized model for this application but without having to send data to the server (thus saving training time and communication bandwidth requirements).\n\nCons:\n- No actual research contribution since nothing new is proposed in this paper.\n- While the training time and communication bandwidth savings are a good validation, this is not surprising since Federated Learning has been shown to have this benefit for many applications. \n\n========== UPDATE AFTER REBUTTAL ===========\nI have read the author's response. While the case study for industrial applications is important, it would probably be much more impactful if the same study was done on a much larger/realistic scale. For instance, right now it appears that each edge vehicle gets an already available dataset for federated learning, which may have been cleaned and preprocessed properly. For claiming a real industrial deployment/importance, it would have been great if the study was conducted with vehicles receiving real-time data from real vehicles which is prone to be extremely noisy (although the reviewer is not sure if this would be possible for regulatory reasons (e.g., if such learning experiments would be safe enough on real autonomous vehicles as these applications are safety-critical)). Currently, the paper neither has significant enough contributions from novelty side, nor from industrial deployment angle. Hence, as such, the paper cannot be accepted. Perhaps more application-oriented conferences maybe more suitable for this kind of work. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the use of federated learning to the application of steering wheel prediction for autonomous driving. While the application is new and interesting, the reviewers felt that the approach and results were mostly empirical. I suggest that the authors improve the conceptual/algorithmic contribution of the paper in a revised draft. Another suggestion is to include a better explanation of hyper-parameter optimization used in the experiments. I hope that the reviewers' constructive comments will help the authors revise the draft adequately for submission to a future venue!
Decision: Reject