Smart Multi-tenant Federated Learning
Authors: Weiming Zhuang
Yonggang Wen
Shuai Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: federated learning
multi-tenant federated learning

Abstract: Federated learning (FL) is an emerging distributed machine learning method that empowers in-situ model training on decentralized edge devices. However, multiple simultaneous training activities could overload resource-constrained devices. In this work, we propose a smart multi-tenant FL system, MuFL, to effectively coordinate and execute simultaneous training activities. We first formalize the problem of multi-tenant FL, define multi-tenant FL scenarios, and introduce a vanilla multi-tenant FL system that trains activities sequentially to form baselines. Then, we propose two approaches to optimize multi-tenant FL: 1) activity consolidation merges training activities into one activity with a multi-task architecture; 2) after training it for rounds, activity splitting divides it into groups by employing affinities among activities such that activities within a group have better synergy. Extensive experiments demonstrate that MuFL outperforms other methods while consuming 40% less energy. We hope this work will inspire the community to further study and optimize multi-tenant FL.

Under review as a conference paper at ICLR 2023
SMART MULTI-TENANT FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL) is an emerging distributed machine learning method that empowers in-situ model training decentralized edge devices. However, multiple simultaneous training activities could overload resource-constrainee devices. In this work, we propose a smart multi-tenant FL system, MuFL, effectively coordinate and execute simultancous training activities. We first formalize the problem of multi-tenant FL, define multi-tenan FL scenarios, and introduce a vanilla multi- tenant FL system that trains activities sequentially to form baselines. Then, we propose two approaches to optimize multi-tenant FL: 1) activity consolidation merges training activities into one activity with a multi-task architecture; 2) after training for rounds, activity splitting divides into groups by employing affinities among activities such that activities within a group have better synergy. Extensive experiments demonstrate that MuFL outperforms other methods while consuming 40% less energy. We hope this work will inspire the community to further study and optimize multi-tenant FL.
INTRODUCTION
Federated learning (FL) (MeMahan al. 2017)) has attracted considerable attention it enables privacy-prece privacy-preserving distributed model training among decentralized devices. It empowering growing numbers of applications in both academia and industry, such as Google Keyboard (Hard et al. 2018 medical imaging analysis (Li et al. 2019] Sheller et al., (2018), and autonomous vehicles (Zhang et al. 2021a Posner et al.] 2021). Among them, some applications contain multiple training activities for different tasks. For example, Google Keyboard includes query suggestion (Yang et al. 2018), emoji prediction (Ramaswamy et al., 2019), and next-world prediction (Hard et al. (018]) 8); autonomous vehicles relates multiple computer vision (CV) tasks, including lane detection, object detection, and semantic segmentation (Janai et al. 2020).
However, multiple simultaneous training activities could overload edge devices (Bonawitz et al. 2019). Edge devices have tight resource constraints, whereas training deep neural networks for the aforementioned applications is resource-intensive. As result, the majority of edge devices can only support one training activity at a time (Liu et al.] 2019); multiple simultaneous federated learning activities the it is important to navigate solutions to well coordinate these training activities.
device could overwhelm its memory, computation, and power capacities. Thus,
plethora of research on FL considers only one training activity in an application. Many studies are devoted to addressing challenges including statistical heterogeneity (Li et al. 2020 Wang et al.. 2020a system heterogeneity (Chai et al., 2020 Yang al., (02T)) communication efficiency
KKrrmireddy et al.) 2020; (Zhu et al. 2021). and privacy issues (Bagdasaryan et al. 2020 Huang et al. 2021). common limitation that they only focus on one training activity. but applications like Google Keyboard and autonomous vehicles require multiple training activities for different targets (Yang et al. 20088] Raaaasamy et al., 2019). Multi-tenancy of FL. system is designed by Bonawitz al. (2019) to prevent simultancous training activities from overloading devices. However, it mainly considers differences among training activities, neglecting potential synergies.
In this work, we propose smart multi-tenant federated learning system, MuFL, to efficiently coordinate and execute simultaneous training activities under resource constraints by considering both synergies and differences among training activities. We first formalize the problem of multi- tenant FL and define four multi-tenan FL scenarios based on two variances in Section 3; 1) whether all training actívities are the same type of application, e.g., CV applications; 2) whether all clients

---

Under review as a conference paper at ICLR 2023
support all training activities. Then, we define vanilla multi-tenant FL system that supports all scenarios by training activities sequentially. Built on it, we further optimize the scenario, where all training activities are the same type and all clients support all activities, by considering both synergies and differences among activities in Section 4 Specifically, we propose activity consolidation to merge training activities into one activity with multi-task architecture that shares common layers and has specialized layers for each activity. We then introduce activity splitring to divide the activity into multiple activities based on their synergies and differences measured by affinities between activities.
We demonstrate that MuFL reduces the energy consumption by over 40% while achieving superior performance to other methods via extensive experiments on three different sets of training activities in Section 5 We believe that MuFL is beneficial for many real-world applications such as autonomous vehicles, voice assistance systems, and roboties (more examples in Appendix A). We summarize our contributions as follows:
We formalize the problem of multi-tenant FL and define four multi-tenant FL scenarios. To the best of our knowledge, we are the first work that investigates multi-tenant FL in-depth.
We propose MuFI., smart multi-tenant federated learning system to efficiently coordinate and execute simultaneous training activities by proposing activity consolidation and activity splitting to consider both synergies and differences among training activities.
We establish baselines for multi-tenant FL and demonstrate that MuFL elevates performance with significantly less energy consumption via extensive empirical studies.
RELATED WORK
In this section, first review the concept of multi-tenancy in cloud computing and machine learning. Then, we provide a literature review of multi-task learning and federated learning.
Multi-tenancy of Cloud Computing and Machine Learning Multi-tenancyy has been an important concept in cloud computing. It refers to the software architecture where a single instance of software serves multiple users (Chong & Carraro| 2006 |ehllng et al. 2010). Multi-tenant software archi- tecture is one of the foundations of software as service (SaaS) applications (Mietzner et al. 2008
Cai et al. 2013). Recently, researchers have adopted this idea to machine learning (especially deep learning) training and inference. Specifically, some studies investigate how to share GPU clusters among multiple users to train deep neural networks (DNN) (Jeon al.) 2019 Zhao et al. 2020 Lao et al., 2021), but these methods are for GPU clusters that have enormous computing resources, which are inapplicable to edge devices that have limited resources. Targeting on-device deep learning, some researchers define multi-tenant as processing multiple computer vision (CV) applications for multiple concurrent tasks (Fang et al. 2018 Jiang et al., 2018 However, they focus on the multi-tenant on-device inference rather than training. On the contrary, we focus on multi-tenant federated learning (FL) training on devices, where the multi-tenancy refers to multiple concurrent FL training activities.
Multi-task Learning Multi-task learning is a popular machine learning approach to learn models that generalize on multiple tasks (Thrun) [995] Zhang& & Yang 2021). A plethora of studies investigate parameter sharing approaches that share common layers of similar architecture (Caruana 1997]
Figen & Fergus] 2015 Bilen & Vedaldi 2016; Nekrasov et al. (219)). Besides, many studies employ new techniques to address the negative transfer problem (Kang et al. [201]] Zhao et al. 20188) among tasks, including soft parameter sharing (Duong et al. 2015]] Misra et al. (016), neural architecture
search (Lu al. 2017 HHang al. 2018 Vandenhende al. 2019
Guo Guo al., 2020 Sun et al.
2020), and dynamic loss reweighting strategies (Kendall et al. 2018 Chen et al. 2018 Yu et al.
2020). Instead of training all tasks together, task grouping trains only similar tasks together. The
early works of task grouping (Kang et al. 2011 Kumar & Daumé 2012) are not adaptable to DNN. Recently, several studies analyze the task similarity (Standley et al. (020)) and task affinities (Fifty iet al., 2 for task grouping. In this work, we adopt the idea of task grouping to consolidate and split training activities. The state-of-the-an task grouping methods (Standley et al. 2020 Fifty eet al. 202T), however, unsuitable for our scenario because they focus on the inference efficiency, bypassing the intensive computation on training. Thus, we propose activity consolidation and activity splitting to group training activities based their synergies and differences.
Federated Learning Federated learning emerges as a promising crivacy-peseevve distributed machine learning technique that uses central server coordinate multiple decentralized clients to
2

---

Under review as a conference paper at ICLR 2023
train models (McMahan et al. 2017 Kairouz al. 2021). The majority of studies aim to address the
challenges of FL, including statistical heterogeneity (Li et al. 2020 Wang et al. 2020a b Zhuang
ct al. 2020; yuyang deng al. 2021 Zhang al. [2021b), system heterogeneity (Chai et al.
2020; Yang et al. 2021), 1, communication efficiency (McMahan et al. 2017, Konečný et al. 2016
Karimireddy et 2020 Zhu et al. 202T), and privacy concerns (Bagdasaryan et al. |0200] Huang
et al., (021). Among them, federated multi-task learning (Smith et al. 2017: Marfog et al. 2021)
an emerging method to learn personalized models to tackle statistical heterogeneity. However, these personalized FL. methods mainly focus on training one activity of an application in a client. Multi-tenan tulti-tenant FL that handles multiple concurrent training activities is rarely discussed; the prior work (Bonawitz et al. 2019) mainly considers the differences among training activities. In this work, we optimize multi-tenant FL by further considering their synergies by splitting activities into groups. Our problem is also fundamentally different from clustered FL. (Ghosh et al. 2020; Ouyang et al. 2021; They group models of the same training activity, whereas we group training activities.
PROBLEM SETUP
This section provides preliminaries of federated learning (FL.), presents the problem definition of multi-tenant FL, and classifies four multi-tenant FL scenarios. Besides, we introduce vanilla multi-tenant FL system supports for all scenarios.
3.1 PRELIMINARIES AND PROBLEM DEFINITION
In the federated learning setting, the majority of studies consider optimizing the following problem:
K
K
min f(w) = Σ [psfuww>= = En (wr WERd
(I)
k=1
where w is the optimization variable, K is the number of selected clients to execute training, fu (w) is the loss function of client k. Ps: the weight of client & in model aggregation, and & is the training data sampled from data distribution Dr of client k. FedAvg (McMahan et al. (017) is popular federated learning algorithm, which sets pk to be proportional to the dataset size of client k.
Equation [ illlsstrates the objective of single training activity in FL, but in real-world scenarios, multiple simultaneous training activities could overload edge devices. We further formalize the problem of multi-tenant FL as follows.
In multi-tenant Fl., server coordinates a set of clients C to execute a set of FL training activities A = {a1, .2.......................... ). It obtain set of parameters of models W {wi, ww0,,,......44 w..). where each model wi is for activity a, By defining M(a;; w;) as performance measurement of each training activity Og multi-tenant FL aims to maximize the performance of all training activities EMnMM(as: w.). under the constraint that each client has limited memory budget and computation budget. These budgets constrain the number of concurrent training actvitities ng client k. Besides, as devices have limited battery life, we would like to minimize the energy consumption and training time to obtain W from training activities A.
3.2 MULTI-TENANT FL SCENARIOS
We classify multi-tenant FL into four different scenarios based on variances in two aspects: 1) whether all training activities in A are the same type of application, e.g., computer vision (CV) applications or natural language processing (NLP) applications; 2) whether all clients in C support all training activities in A. We depict these four scenarios in Figure 6 in Appendix A and describe them below.
Scenario 1 Va; E A. a4 is the same type of application; Va, A. Yck C supports ag For example, autonomous vehicles (clients) support the same sets of CV applications, such as object detection and semantic segmentation. Thus, they support training activities of these applications.
3as E A. a; is a different type of application; Va, E A, Vck E C supports a; For example, Google Keyboard has different types of applications, including recommendation (query suggestion (Yang et al.) (2018) and NLP (next-world prediction (Hard et al. 2018)). Mobile phones (clients) with Google Keyboard support these applications together with all related training activities.
Scenario 2
3

---

Under review as a conference paper at ICLR 2023
Server
Server
Scheduling
Activity Consolidation
2.Scheduling
Training Activities
Training Activities
3. Training naa
Activity Splitting
Training
2:1 Execution
2.2 Aggregation
3.1
3.2 Aggregation
Partispants
Participaniss
Client Pool
Client Pool
Training types
Status
New training activity
Scheduled
Trainad
One
Allii-one
Grouped
(a) Vanilla Multi-tenant FL
(b) Smart Multi-tenant FL
Figure 1: The architectures of proposed multi-tenant federated learning (FL) systems. The vanilla multi-tenant FL system (a) employs a scheduler to queue simultaneous training activities and exe- cute them one one. The smart multi-tenant FL system (b) proposes activity consolidation and activity splitting to consider both synergies and differences among training activities, which elevates performance and reduces resource consumption.
Scenario 3 Vo, E A, a; the same type of application; Va; E A, 3e, E C does not support og. For example, survelliance cameras (clients) in parking lots could support CV applications, but cameras in different locations may support different applications, e.g., counting open-parking spots, tracking parking duration, or recording fender benders.
Scenario 4 3a, E A, a₁ is a different type of application; Ya, E A, Ecg E C does not support ag For example, browsers (clients) could leverage users' browsing history to support ranking (Hartmann et al.] (2019) and news recommendation (Minto et al. 202I), which are different applications. Users may opt-out of recommendations, resulting in not all browsers supporting all training activities.
The application determines the multi-tenant FL. scenario. We next introduce a vanilla multi-tenant FL
that supports all these scenarios as our baseline.
3.3 VANILLA MULTI-TENANT FL.
Figure la presents the architecture of vanilla multi-tenant FL system. It prevents overloading and congestion of multiple simultaneous training activities by scheduling them to execute one by one. Particularly, we scheduler queue training activities in the server (e.g., First in, First out). In each round, the server selects K clients from the client pool to participate in training. The number of simultaneous training activities depend on the computational resources the selected clients. In this study, we assume that each client can execute one training activity at time (nę = 1). This is realistic assumption for the majority of current edge devices. FL system executes training activities sequentially.
result, the vanilla multi-tenant
The vanilla multi-tenant system supports the four multi-tenant FL scenarios. From the perspective of the type of application, it can handle different application types of training activities as each training activity is executed independently. From the perspective of whether clients support all training activities, each training activity select clients that support the activity to participate in training. Despite its comprehensiveness, only considers differences among training activities, neglecting their potential synergies. In contrast, our proposed MuFL considers both synergies and differences among training activities to further optimize the Scenario 1.
Edggss devices, e.g., NVIDIA Jetson TX2 and AGX Xavier, have only GPU; GPU virtualization (Hong 2017) that enables concurrent training the same GPU currently mainly for the cloud stack.
4

---

Under review as a conference paper at ICLR 2023
4 SMART MULTI-TENANT FL
In this section, we introduce the smart multi-tenant FL system, MuFL. We start by providing an overview of MuFL. Then, we present two important components of MuFL, activity consolidation and activity splitting, to consider both synergies and differences among simultaneous training activities.
Figure dedepicss the architecture and training processes of MuFL. It contains server to coordinate training actívities and pool of clients to execute training. MuFl. optimizes the Scenario 1 discussed previously with the following steps: 1) The server receives training activities A {(1, 022, 1.2....... (n)
to train models W {wi, w2,...., Wn ) and consolidates these activities into an all-in-one training activity ag: 2) The server schedules ag to train: 3) The server select K clients from the client pool to execute ogg iteratively through FL process for Rg rounds; 4) The server splits the all-in-one activity og into multiple training activity groups (A1..2.....
where each group trains nonoverlapping subset of W; the number of groups can be determined by the inference budget for the number of concurrent models. 5) The server iterates step 2 and 3 to train Aj. We summarize MuFL in Algorithm Ii Appendix and introduce the details of activity consolidation and activity splitting next.
4.1 ACTIVITY CONSOLIDATION
Focusing on optimizing the Scenario of multi-tenant FL., we first propose activity consolidation to consolidate multiple training activities into an all-in-one training activity, as illustrated in the first step of Figure [b] In Scenario all training activities are the same type of application and all clients support all training activities. Since training activities A = {a1, ..,,...,0 .., are of the same type, e.g.. CV or NLP, models W = fwi, w2 ............ .,) could share the same backbone (they share the same encoder but could have different decoders). Thus, we can consolidate A into all-in-one training activity og that trains a multi-task model v = [0.] U [0,, E A}, where 0, is the shared model parameters and 0a, is the specific parameters for training activity og A. The loss function for all-in-one training is C(X, 8,, (0,, = -.. Ca, (X, (x, (0a,))
Activity consolidation leverages synergies among training activities and effectively reduces the computation cost of multi-tenant FL from multiple trainings into a single training. However, simply employing activity consolidation is another extreme of multi-tenant FL that only considers synergies among activities. As shown in Figure 2] all-in-one method is efficient in energy consumption, but it leads to unsatisfactory performance. Consequently, we further propose activity splitting consider both synergies and differences among training activities.
4.2 ACTIVITY SPLITTING
We propose activity splitting to divide the all-in-one activity og into multiple groups after it is trained for certain rounds. Essentially, we aim to split A = {a1, a....... , an into multiple nonoverlapping groups such that training activities within a group have better synergy. Let {Ai, Az.... Am be subsets of A, we aim to find a disjoint set of A, where S (1, 2, ........ mm. S [A], U5e Aj A.
and NgEr Aj 6. Each group Aj trains a model v; = (00) (0,, la; E A, which is a multi-task network when A, contains more than one training activity, where A is the shared model parameters and 0., is the specific parameters for training activity a E A, The core question is how to determine set split these activities considering their synergies and differences.
Inspired by TAG (Fifty et al. 2021) that measures task affinites for task grouping, we employ affinities between training activities for activity splitting via three stages: 1) we measure affinities among activities during all-in-one training; 2) we select the best combination of splitted training activities based on affinity scores; 3) we continue training each split with its model initialized with parameters obtained from all-in-one training. Particularly, during training of all-in-one activity og, we measure the affinity of training activity a; onto aj at time step in each client & with the following equation:
Shit ta = 1 Lo, (X k. 0k.++1 gh.t)
La, (Xth,s, 1ackss.0% oh.e 0
where Lay is the loss function of aj, Kha is batch of training data, and ghat and gla+1 are the shared model parameters before and after updated by as respectively. Positive value of sk.t 0,
5

---

Under review as a conference paper at ICLR 2023
0.68
X4 All-in-on one
x GradNorm
Splits Splits
1,50
Allin--nee FedFrrox
GradNorm
TAG
Leg
0.62
0.6-1
0.66
x FedPPrx
TAG
Splits
HOA
Test Loss
1.41
1.46
1.48
Ours
X One by ane
TAG
HOA
0.58
Ours
by one
1.42
Better
Splits Splits
Splits Splits
0.56
Qur
Ours
1.40
Ours
Ours
Ours
1.38
20 Energy (kWh)
16
24
28
32
12
14
Energy (kWh)
(a) A set of five training activitics: sdrkt
(b) set of nine trianing activities: sdnkterca
Figure 2: Comparison of test loss and energy consumption on two training activity sets: (a) sdnkt and (b) sdnkterca, where each character represents an activity. Compared with all-in-one methods, our method achieves much better performance with slight increases in computation. Moreover, our method achieves the best performance while consuming less energy than the other methods.
means that activity helps reduce the loss of oj. This equation measures the affinity of one time- step of one client. approximate affinity scores for each round by averaging the values over T time-steps in local epochs and K selected clients: Su,-aa = KFT E Σe= Σ
Di=1 aa
Sk.t
where T is total time steps determined by the frequency of calculating Equation 2 e.g., f 2 means measuring the affinity in each client in every five batches.
These affinity scores measure pair-wise affinities between traininig activities. We next use them to calculate total affinity scores of a grouping with Σi= Sa,, where S., is the averaged affinity score onto each training activity. For example, in grouping of two splits of five training activities {fa (a1,a))) {as, a4, a,)), where {ai, } is one split and {as, as. as is another split. The affinity score onto is S.n, = Saz-aa and the affinity score onto 03 is Sa, = (Saa + Sas-+aa)/ 1/2 Consequently, we find the set I with I elements for subsets of A that maximize Ei= Sa.. where T| defines the number of elements.
We would like to further highlight the differences between method and TAG (Fifty et al.) 2021 Firstly, TAG focuses on inference efficiency, thus it allows overlapping task grouping that could train one task multiple times. In contrast, our focus is fundamentally different; we focus on training efficiency and consider only nonoverlapping activity splitting. Secondly, TAG is computation- intensive for higher numbers of splits, e.g., fails to produce results of five splits of nine tasks in week, whereas we only need seconds of computation. Thirdly, TAG rules out the possibility that a group contains only one task as it sets Sa,-ta 1e-6 which is much smaller than scores of other groupings. Besides, Sa,+o; calculated from Equation 2 is also not desirable; it always results in group containing only one task as its value is much larger (could be 10x larger) than scores of other groupings. To overcome these issues, propose new method to calculate this value:
(Sa. Saa
S o. O0
Σ JEN
2n 2
where N' (1, 2,.,,....n] The intuition is that it measures the normalized affinity of activity a; to other activities and other activities to ag Fourthly, we focus multi-tenant FL, thus, we further aggregate affinity scores over K selected clients. Fifthly, TAG trains each set A, from scratch, whereas we initialize their models with the parameters obtained from all-in-one training.
5 EXPERIMENTS
We evaluate the performance and resource usage MuFL and design our experiments to answer the following questions: 1) How effective is activity splitting approach? 2) When to split the training activities? 3) Is it beneficial to iteratively split the training activities? 4) What is the impact of local epoch and scaling up the number of selected clients in each training round?

---

Under review as a conference paper at ICLR 2023
Table 1: Performance (test loss) comparison of our method with the optimal and worst splits. Our method achieves the best performance, indicating the effectiveness of our activity splitting method.
Activity
Train from Scratch
Train from Initialization
Splits
Ours
Set
Optimal
Worst
Optimal
Worst
2
0.578 + 0.015
0.622 = 0.007
0.685 + 0.010
0.595 = 0.008 0.595 + 0.004
sdnkt
0.555
0.008
0.585 + 0.026
0.674 + 0.022
0.560 + 0.006
0.578 + 0.006
2
1.039
0.024
1.070 + 0.013
1.312 + 0.065
1.048 + 0.024
1.068 + 0.037
erckt
3
1.015 + 0.018
1.058 + 0.029
1.243 + 0.099
1.020 + 0.012
1.052 + 0.026
We construct the Scenario / of multi-tenant FL scenarios using Taskonomy dataset (Zamir al. 2018), which is a large computer vision dataset of indoor scenes of buildings. We run experiments with N = 32 clients, where each client contains dataset of a building to simulate the statistical heterogeneity in FL. Three sets of training activities are used to evaluate the robustness of MuFL: sánkt, erckt, and sdnkterca, each character represents an activity, e.g. s represents semantic segmentation. We measure the statistical performance of an activity set using the sum of test losses of individual activities. By default, we use K" = 4 selected clients and E = local epoch for each round of training. More experimental details are provided in Appendix B
Experiment Setup
5.1 PERFORMANCE EVALCATION
We compare the performance, in terms of test loss and energy consumption, among the following methods: 1) one by one training of activities (i.e., the vanilla multi- FL); 2) all-in-one training of activities (i.e., using only activity consolidation): 3) all-in-one training with multi-task optimization (GradNorm (Chen et al. 2018)) and federated optimization (FedProx (Li al. (020)); 4) estimating higher-order of activity groupings from piir-wise activities performance (HOA (Standley et al. 2020)); 5) grouping training activities with only task affinity grouping method (TAG (Fifty et al.
2021 ) 6) MuFL. with both activity consolidation and activity splitting. Carbontracker (Anthony
al. 2020 is used measure energy consumption and carbon footprint (provided in Appendix O
Figure 2 compares performance of the above methods on activity sets sdnkt and adnkterca. The methods that achieve lower test loss and lower energy consumption are better. At the one extreme, all-in-one methods (including GradNorm) consumes the least energy, but their test losses the highest. Simply applying federated optimization, FedProx, can hardly improve performance, especially on sdnkterca. At the other extreme, HOA achieves comparable test losses on three or four splits of sdnkt, but it demands high energy consumption (~ 4 6x of ours) to compute pair-wise activities for higher-order estimation. Although training activities one by one and TAG present a good balance between test loss and energy consumption, MuFL is superior in both aspects; it achieves the best test loss with ~40% and ~50% less energy consumption and sdnkterca, respectively. Additionally, more splits of activity in the activity splitting lead to higher energy consumption, but it could help further reduce test losses. We do not report HOA for activity set sdnkterca due to computation constraints We omit to report the running time as is hidden under the metric of energy consumption; higher energy consumption implies longer training time. We provide more details of these experiments and results of activity set e erckt in Appendix
activity set sdnkt
5.2 How EFFECTIVE IS OUR ACTIVITY SPLITTING APPROACH?
We demonstrate the effectiveness of our activity splitting approach by comparing it with the possible optimal and worst splits. The optimal and worst splits are obtained with two steps: 1) we measure the performance over all combinations of two splits and three splits of an activity set by training them from scratch 2) we select the combination that yields the best performance as the optimal split and the worst performance as the worst split.
Table compares the test loss of MuFL with the optimal and worst splits trained in two ways: 1) training each split from scratch; 2) training each split the same way as our activity splitting initializing models with the parameters obtained from all-in-one training. On the one hand, training
'HOA computes at least 36 pairs of activities (~720 GPU hours), consuming 122 more energy than MuFl.. There are fifteen and twenty-five combinations of two and three splits, respectively, for of five activities.
7

---

Under review as a conference paper at ICLR 2023
2.50
1.36
0.16
Affmiiy
1.92
1.28
11.68
Alfini
0.12
0,088
0.64
0.34
20.04
0.00
0.000
0.00
Training Round
Training Round
Training Round
(a) Affinities to activity d
(b) Affinities to activity
(c) Affimities activity
Figure 3: Changes of affinity scores of one activity to the other activity set sdnkterca. Activities and r have high inter-activity scores. The trends affinities emerge at the early stage training.
0.05
0.70
2 Spliss -
3 Splits - 4 Spliss
1.14
1.06
1.10
-
2 Splis
3 Splis
4 Splts
1.55
1.50
1.60
Sphts
Splita
5p
6 0.60
Tes:
Tes:
1.45
1.10
0.55
1.35
10
30
80
90
10
20
30
50
80
90
30
40
30
80
70
90
All-in-ne
Rounds (R))
All-in-one Rounds [R)
All-in-one Rounds (Ru)
(a) Training activity set: sdnkt
(h) Training activity set: erckt.
(c) Activity set: sdakterca
Figure 4: Performance comparison of training all-in-one activities for different Ro rounds. Fixing the total training rounds R = 100, our method achieves the best performance when Ro {20, 40).
from initialization outperforms training from scratch in all settings. It suggests that initializing each split with all-in-one training model parameters can significantly improve the performance. other hand, our activity splitting method achieves the best performance in all settings, though training from initialization reduces the gaps of different splits (the optimal and worst splits). These results indicate the effectiveness of our activity splitting approach.
the
5.3 WHEN TO SPLIT TRAINING ACTIVITIES?
We further answer the question that how many Ro rounds should we train the all-in-one activity before activity splitting. is determined by two factors: 1) the rounds needed to obtain affinity scores for reasonable activity splitting; 2) the rounds that yield the best overall performance.
Affinity Analysis We analyze changes in affinity scores over the course of training to show that early-stage affinity scores acceptable for activity splitting. Figure SeggureBrressee presents the affinity scores of different activities to one activity on activity set sdnkterca. Figure [3 and (3bjindicat that activity and activity have high inter-activity affinity scores; they are divided into the same group result. In contrast, both d and r have high affinity score to activity s in Figure [c] but not vice versa. These trends emerge the early stage of training, thus, we employ the affinity scores of the tenth round for activity splitting by default; they are effective in achieving promising results as shown in Figure 2 and Table I
We provide more affinity scores of other activities in Appendix C
The Impact of R2 Rounds Figure compares the performance of training Rg for 10 to 90 rounds before activity splitting. Fixing the total training round
100, we train each split of activities for R = R - Ro rounds. The results indicate that MuFL achieves the best performance when Rg (20, 30, 40) rounds. Training the all-in-one activity for enough rounds helps utilize the benefits and synergies of training together, but training for too many rounds almost suppresses the benefits of considering differences among activities. We suggest training Ro for [20, 40] that strikes a good balance between these two extremes and consider other mechanisms to determine Rg in future works.
5.4 HIERARCHICAL SPLITTING
This section evaluates an alternative activity splitting strategy. In activity splitting, we can divide the all-in-one training activity into 3.... splits. As shown in Figure more splits lead to better performance with slightly higher energy consumption in the five-activity set, but the trend is not straightforward in the nine-activity set. Apart from setting the number of splits directly, MuFL can split the training activity into more splits adaptively via two steps: 1) dividing the all-in-one activity into two splits and training each one for R rounds; 2) further dividing one of them to two splits and train these three activities for R2 rounds. We term the adaptive process as hierarchical splitting.
8

---

Under review as a conference paper at ICLR 2023
Table Performance of hierarchical splitting on three activity sets sänkt, erckt, and sdnkterca. Hierarchical splitting outperforms two splits and achieves similar performance to three splits with less energy (kWh) consumption.
sdnkt
erckt
sänkterca
Method
Energy
Test Loss
Energy
Test Loss
Energy
Test Loss
Two Splits
4.9 - 0.3
0.578 + 0.015
6.7 + 0.2
1.039 + 0.024
6.0 + 0.1
1.445 = 0.021
Three Splits
5.4 14::00 0.7
0.555 + 0.008
7.2 0.2
1.015 1 0.018
6,6 # 0.4
1.391 + 0.030
Hierarchical
5.3 - 0.4
0.563 + 0.007
6.9 + 0.2
1.022 + 0.020
6.5 + 0.3
1.403 + 0.024
2 compares the performance of hierarchical splitting (3 splits) with directly splitting to multiple splits on three activity sets. We use Rg = 30, R 40, and R2 30 for activity sets sdnkt and erckt, and Ro 30, Rı 20, and Rg = 50 for sdnkterca. Hierarchical splitting effectively reduces test losses of two splits and achieves comparable performance to three splits with less energy consumption. These results suggest that hierarchical splitting can be an alternative method of activity splitting. Additionally, these results also demonstrate the possibility other activity splitting strategies be considered in future works.
5.5 ADDITIONAL ANALYSIS
This section analyzes the impact of local epoch E and the number of selected clients K in FL using all-in-one training. We report the results of activity set sdnkt here and provide more results in Appendix
Test Loss
0.7
Test Loss
0.9
0.8
0.6
0.7
0.6
0.5
2
8 16
Impact Local Epoch E Local epoch defines the num- ber of epochs each client trains before uploading training updates to the server. Figure [a compares test losses of local epochs E = {1,2, 2, 5, 10). Larger E could lead to better performance with higher computation (fixed train- effective when increasing E = 5 to E = 10. It suggests the limitation of simply increasing computation with larger E' in improving perfor- mance. Note that MuFL (Table I achieves better results than E with 5x less computation.
E
(a) Impact of E
(b) Impact of K
Figure 5: Analysis of the impact of (a) local epoch E and (b) the number of se- lected clients K on activity set sdnkt. Larger E and K could reduce losses with more computation, but the benefit decreases computation increases.
ing round R 100), but it
Impact of The Number of Selected Clients K Figure 5b compares test losses of the number of selected clients K = {2, 4, 8, 16) in each round. Increasing the number of selected clients improves the performance, but the effect becomes marginal as K increases. Larger K can also be considered as using more computation in each round. Similar to the results of the impact of E, simply increasing computation significance of MuFL that increases performance with slightly more computation. We use K = 4 by default for experiments and demonstrate that MuFL is also effective on K = 8 in Appendix C
only improve performance to a certain extent. It also shows the
CONCLUSIONS
In this work, we propose smart multi-tenant federated learning system to effectively coordinate and execute multiple simultaneous FL training activities. In particular, we introduce activity consolidation and activity splitting to consider both synergies and differences among training activities. Extensive empirical studies demonstrate that our method is effective elevating performance and significant in reducing energy consumption and carbon footprint by more than 40% which are important metrics to our society. We believe that multi-tenant FL will emerge and empower many real-world applications with the fast development of FL. We hope this research will inspire the community to further work on algorithm and system optimizations of multi-tenan FL. Future work involves designing better scheduling mechanisms to coordinate training activities. Client selection strategies also be considered to optimize resource and training allocation, and extend our optimization approaches to other multi-tenant FL scenarios.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
To facilitate reprodueibility, we provide basic experimental setups in Section and include more details about the dataset, implementation details, and hyperparameters in Appendix B We also provide the algorithm of MuFL in Algorithm Besides, the implementation codes will be open-sourced in the future.
REFERENCES
Lasse Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. ICML Workshop on Challenges in Deploying and monitoring Machine Learning Systems, July 2020. arXiv:2007.0305.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948 PMLR, 2020.
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. Advances in neural information processing systems, 29, 2016.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, et Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems, 1: 374-388, 2019.
Hong Cai, Berthold Reinwald, Ning Wang, and Chang Jie Guo. Saas multi-tenancy: Framework, technology, and case study. In Cloud Computing Advancements in Design, Implementation, and Technologies, pp. 67-82. IGI Global, 2013.
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
Zheng Chai, Ahsan Ali, Syed Zawad. Stacey Truex, Ali Anwar, Nathalic Baracaldo, Yi Zhou, Heiko Ludwig, Feng Yan, and Yue Cheng. Tifl: A tier-based federated learning system. In Proceedings of the 29th International Symposium High High-Performance Parallel and Distributed Computing. pp. 125-136, 2020.
Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning. pp. 794-803. PMLR, 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Frederick Chong and Gianpaolo Carraro. Architecture strategies for catching the long tail. MSDN Library, Microsoft Corporation, 910, 2006.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in neural network parser. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), pp. 845-850, 2015.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE international conference computer vision, pp. 2650-2658, 2015.
Biyi Fang, Xiao Zeng, and Mi Zhang. Nestdnn: Resource-aware multi-tenant on on-device deep learning for continuous mobile vision. Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pp. 115-127, 2018.
Christoph Fehling, Frank Leymann, and Ralph Mietzner. A framework for optimized distribution of tenants in cloud applications. In 2010 IEEE 3rd International Conference on Cloud Computing. pp. 252-259. IEFE, 2010.
10

---

Under review as a conference paper at ICLR 2023
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34, 2021.
FLAIROP. Federated learning for robot picking. 2022. URL https://Elairop.com
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586- 19597, 2020.
Pengsheng Guo, Yuen-Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, pp. 3854-3863, PMLR, 2020.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint 1rXiv: 1811603 2018.
Florian Hartmann, Sunah Suh, Arkadiusz Komarzewski, Tim D Smith, and Ilana Segall. Federated learning for ranking browser history suggestions. arXiv preprint arXiv: 1807, 2019.
Cheol-Ho Hong, Ivor Spence, and Dimitrios S Nikolopoulos. Gpu virtualization and scheduling methods: A comprehensive survey. ACM Computing Surveys (CSUR), 50,3):1-37,200 2017.
Siyu Huang, Xi Li, Zhi-Qi Cheng, Zhongfei Zhang, and Alexander Hauptmann. Gnas: greedy neural architecture search method for multi-attribute learning. In Proceedings of the 26th ACM international conference on Multimedia, pp.
2018.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34, 2021.
Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger, et al. Computer vision for autonomous vehicles: Problems, datasets and state the art. Foundations and Trends in Computer Graphics and Vision, 12(1-3):1-300 2020.
Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. Analysis of Large-Scale Multi-Tenant GPU clusters for DNN training workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 947-960, Renton, WA, July 2019. USENIX Association. ISBN 97--1-991333333
URL www usenix.org/
Angela H Jiang. Daniel L-K Wong. Christopher Canel, Lilia Tang, Ishan Misra, Michael Kaminsky, Michael A Kozuch, Padmanabhan Pillai, David G Andersen, and Gregory R Ganger. Mainstream: Dynamic them--Shrrigg for (Multi-Tenant)) video processing. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pp. 29-42, 2018.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends in Machine Learning. 14(1-2):1-210, 2021.
Zhuoliang Kang. Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In /CML, 2011.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018
11

---

Under review as a conference paper at ICLR 2023
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtarik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. In NIPS Workshop Private Multi-Part Partt Machine Learning, 2016.
ars/abs/////
1610. 05492
Abhishek Kumar and Hal Daumé. Learning task grouping and overlap in multi-task learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1723-173233 2012.
ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift. ATP: In-network aggregation for multi-tenant learning. In /8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pp. 741-761. USENIX Association, April 2021. ISBN 978-1-939133-21-22 URL https: /wwp///www..... usenix org/ congg//oneeeeccc///
nsdi21/presentat 1ao
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
Wenqi Li, Fausto Milletari, Daguang Xu. Nicola Ricke, Jonny Hancox, Wentao Zhu, Maximilian Baust, Yan Cheng, Sébastien Ourselin, M Jorge Cardoso, et al. Privacy-preserving federated brain tumour segmentation. International Workshop on Machine Learning in Medical Imaging, pp- 133-141. Springer, 2019.
Jie Liu, Jiawen Liu, Wan Du, and Dong Li. Performance analysis and characterization of training deep learning models on mobile device. In 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), pp. 106-515. IEEE, 2019.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5334-5343, 2017.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under mixture of distributions. Advances in Neural Information Processing Systems, 34, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effiiieell learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1288 PMLR, 2017.
Ralph Mietzner, Frank Leymann, and Mike Papazoglou. Defining composite configurable saas application packages using sca, variability descriptors and multi-tenancy patterns. In 2008 third international conference on Internet and web applications and services, pp. 156-161. IEEE, 2008
Lorenzo Minto, Moritz Haller, Benjamin Livshits, and Hamed Haddadi. Stronger privacy for federated collaborative filtering with implicit feedback. In Fifteenth ACM Conference on Recommender Systems, pp. 1442-500220 2021.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3994-4003, 2016.
Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, and Ian Reid. Real-time joint semantic segmentation and depth estimation using asymmetric annotations. In 2019 International Conference on Robolies and Automation (ICRA), pp. 7101-7107. IEEE, 2019.
Xiaomin Ouyang, Zhiyuan Xie, Jiay Zhou, Jianwei Huang, and Guoliang Xing. Clusterfl: a similarity-awe federated learning system for human activity recognition. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, pp. 54-66, 2021.
12

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jason Posner, Lewis Tseng, Moayad Aloqaily, and Yaser Jararweh. Federated learning in vehicular networks: opportunities and solutions. IEEE Network, 35(2):152-159, 2021.
David Qiu, Yanzhang He, Qiujia Li, Yu Zhang, Liangliang Cao, and Ian McGraw. Multi-task learning for end-to-end asr word and utterance confidence with deletion prediction. arXiv preprint arXiv:2104.1288 128700 2021.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated learning for emoji prediction in mobile keyboard. arXiv preprini arXiv: 1906.0439, 2019,
Micah J Sheller, G Anthony Reina, Brandon Edwards, Jason Martin, and Spyridon Bakas. Multi- institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation. In International MICCAI Brainlesion Workshop, pp. 92-104. Springer, 2018.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning, Advances neural information processing systems, 30, 2017.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pp. 9120-9132. PMLR, 2020.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33: 6728-87400 2020.
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in neural information processing systems, 8, 1995.
Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint wxXhv1904.009220 2019.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- erated learning with matched averaging. International Conference on Leurning Representations, 2020a.
net/ forum? wtt/:/ooummmaaaam/o -id: 1
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.0748) 2020b.
Chengxu Yang, Qipeng Wang, Mengwei Xu, Zhenpeng Chen, Kaigui Bian, Yumxin Liu, and Xuanzhe Liu. Characterizing impacts of heterogeneity federated learning upon large-scale smartphone data. In Proceedings of the Web Conference 2021, pp. 935-946, 2021.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv: /8/ 8.002003 2018
Jianbo Ye and Arnie
Sen.
How
does
astro
localize itself in an ever-
changing home?
URL https://www. www.wwwwwwwwa........ sci www.ssceeeeenniceeeeeee htw.:oooooaaieee-te-------------
2022.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824 5836, 2020.
yuyang deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. learning, 2021. URL http///pppreiieeeeeeeeeeee www. -eeet//orrumm11=g000==>x00
federated
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 3712-3722, 2018
13

---

Under review as a conference paper at ICLR 2023
Hongyi Zhang, Jan Bosch, and Helena Holmström Olsson. End-to-end federated learning for autonomous driving vehicles. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021a.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Processing Systems, 34, 2021b.
Yu Zhang and Qiang Yang. survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
Hanyu Zhao, Zhenhua Han, Zhi Yang. Quanlu Zhang, Fan Yang, Lidong Zhou, Mao Yang, Francis C.M. Lau, Yuqi Wang, Yifan Xiong, and Bin Wang. HiveD: Sharing a GPU clus- ter for deep learning with guarantees. In 14th USENIX Symposium Operating Systems Design and Implementation (OSDI 20), pp. 515-532. USENIX Association, November 2020.
ISBN 978-1-939113-19-99 URL https:// wwwp:/wwwwwww...ooeem..m......xt otgg/confeernnce//dd20000 Cresentation/zhaoooeaee 2hao
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference Computer Vision (ECCV), pp. 401-416, 2018.
Ligeng Zhu, Hongzhou Lin, Yao Lu, Yujun Lin, and Song Han. Delayed gradient averaging: Tolerate the communication latency for federated learning. Advances in Neural Information Processing Systems, 34, 2021.
Weiming Zhuang, Yonggang Wen, Xuesen Zhang, Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi. Performance optimization of federated person re-identification via benchmark analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 955 -639,, 2020.
14

---

Under review as a conference paper at ICLR 2023
A MULTI-TENANT FL SCENARIOS
We introduce four multi-tenant federated learning scenarios in Section 3 four scenarios with variances in two aspects: 1) whether all training activities the same type of application, e.g., CV applications: 2) whether all clients support all training activities.
Figure depicts these
Clients support
activities
Example: Google Keyboard
Example: Autonomous Vehicle
80800
Clients: Smartphones
Clients: Cars
Type: Recommendation, NLP Training activities: e.g. query suggestion, next-word prediction
Type: Computer vision
Training activitles: e.g., object detection, semantic segmentation
Different Types of Applications
Same Type of
Applications
Example: Browsers
Example: Survelllance Cameras
Parking Lots
Clients: Browsers
Clients: Cameras
WWW
Type: Recommendation, Ranking
Type: Computer vision
Training activities: news/ads
Training activities: e.g.. open spots counting, duration tracking
recomienddttio, rank history
Not all clients
support all activities
Figure 6: Illustration of the four multi-tenant FL scenarios.
Our proposed approach, MuFL, focuses optimizing the performance Scenario 1. Optimizing Scenario can potentially empower plenty of real-world applications. For example, autonomous vehicles relate multiple computer vision (CV) tasks (Janai et al.] 2020), including object detection, tracking, and semantic segmentation; smart city surveillance cameras associate with various CV tasks, such crowd counting, object detection, and person re-identification (Zhuang al. 2020); voice assistant applications like Apple Siri and Google Assistant need multiple automatic speech recognition (ASR) tasks, including word confidence, word deletion, and utterance confidence (Qiu et al. 202T;; household robots like Amazon Astro need to perform multiple CV tasks such as visual odometry tracking, loop-closure detection, and object detection (Ye & Sen| [022 smart- manufacturing robots need several CV tasks such as object detection, object grasp point detection,
and object pose estimation FLAIROP 2022).
B EXPERIMENTAL DETAILS
This section provides more experimental information, including dataset, implementation details, and computation resources used.
Dataset We run experiments using Taskonomy dataset
(Zamir et al. 2018), which is large computer vision
(CV) dataset of indoor scenes of buildings. To facili-
Data Amount
20000
16000
tate reproducibility and mitigate computational require- ments, we use the tiny split of Taskonomy dataset whose size is around 445GB. We select nine CV applications to form three sets of training activities: sünkt, e erckt, sdnkterca. These nine actvities are also used in (Stan- dley et al. 2020). Figure g provides sample images of these nine training activities, as well as the representation of each 3 In particular, we employ indoor images
12000
8000
4000
10
20
30
Client ID
Figure 7: The data amount distribution of each training activity over 32 clients.
'Taskonomy dataset is released under MIT license and be downloaded from their official repository
https: : htgggtttuuu.... atam:Staafoooeeo htt.ii//takkoomoyy
'The meaning of each character in sánkterca are as follows; semantic segmentation, d: depth estimation, n: normals, k: keypoint, t: edge texture, e: edge acclusion, r: reshaping, c: principle curvature, auto-encoder.
15

---

Under review as a conference paper at ICLR 2023
Input Image
Segmentation
(o) Depth Estimation
(d)
Surface Normals
(e) Keypoint
t: Edge
(g) Edge Occlusion
(h) Y: Reshasing
ii c: Principle
① a: Auto-encoder
Figure 8: Sample images of nine training activities corresponding to the input image.
Figure Sample images of five clients, where each client contains indoor scenes of building. These indoor images differ in design, layout, objects, and illumination.
of 32 buildings as the total number of clients N 32; each client contains images of a building to simulate the statistical heterogeneity. On the one hand, clients have different sizes of data. Figure 7 shows the distribution of dataset sizes of an activity of clients. On the other hand, Figure 9 shows sample images of five clients; their indoor scenes vary in design, layout, objects, and illumination.
Implementation Details We implement multi-tenant FL systems in Python using PyTorch (Paszke et al. 2017). We simulate the FL training cluster of NVIDIA Tesla V100 GPUs, where each node in the cluster contains GPUs. each round, each selected client is allocated to a GPU to conduct training: these clients communicate via the NCCL. backend. Besides, we employ FedAvg (McMahan et al. 2017) for the server aggregation. By default, we randomly select 4 clients to train for E local epochs in each round and train for R 100 rounds.
We reference the implementation of multi-task learning from (Standley et al. 2020 official reposi- tory [ for all-in-one training and training of each split after activity splitting. Particularly, the network architecture contains an encoder 6g and multiple decoders fa,: one decoder for a training activity o;. Figure 10 illustrates the network architectures of training actívities before and after activity consolidation and activity splitting. We use the modified Xception Network (Chollet 2017) as the encoder for activity sets sdnkt and erckt and half size of the network (half amount of parameters) for activity set sdnkterca The decoders contain four deconvolution layers and four convolution layers. Each training activity contains loss function. Specifically, semnatic segmentation s uses Cross Entropy loss; surface normals and depth estimation use rotation loss based on L1 loss; keypoint detection, edge occlusion, edge texture, auto encoder, and principle curvature use L1 loss. We refer
'The name of the buildings allensville, beechwood, benevolence, coffeen, collierville. corozal, cosmos, darden, forkland, hanson, hiteman, ihlen. klickitat, lakeville, leonardo, lindenwood, markleeville, marstons, mcdade, merom, mifflinburg, muleshoe, newfields, noxapater, onaga, pinesdale, pomaria, ranchester, shelbyville, stockman, tolstoy, and uvalda.
https:/ / /i thub. http...cm//sttaadle//aaagrrooo/
16

---

Under review as a conference paper at ICLR 2023
Training Activity A
Training Activity B
Training Activity
All-in-Ona Training Activity
Training Activity A'
Training Activity B'
Decoder
Dwoxder
Decoder
Desoder
Decoder
Dectader
Decocer
Activity Splitting
Enoodar
Encoder
Encoder
Consolidation
Enoodar
Erroddrr
Enceder
Input Image
Image
Input Image
Input Image
Input
Figure 10: Illustration of network architectures of training activities in MuFL. Initially, each activity employs an encoder and a decoder. Activity consolidation consolidates these training activities into an all-in-one activity with multi-task architecture. After activity splitting. MuFL divides the all-in-one activity into multiple training activities, each contains an encoder and one multiple decoders.
implementatinn of loss functions from Standley et al., 2020 8 The batch size B = 64 for sdnkt and erckt and B 32 for sdnkterca, These are the maximum batch sizes for one GPU without out-of-memory issues. In addition, we use polynomial learning rate decay (1 R )0.9 to update learning rate in each round with initial learning rate 77 0.1, where is the number of trained rounds and R 100 is the default total training rounds. The optimizer is stochastic gradient descent (SGD), with momentum of 0.9 and weight decay 1e-4
Implementation of Compared Methods We tune the hyperparameter 0.004 for the proximal term FedProx (Li et al. 2020). GradNorm (Chen al. 2018) implementation is adopted from
(Standlyy et al. 2020 Fifty et 2021 with default a = 1.5 and TAG (Fifty et al.) (021)) implemen- tation adopted from their official repository Next, we provide the details of how we compute the
results of HOA (Standley et al. 2020) and TAG (Fifty et al. (021).
HOA (Standley al. 2020) needs to compute test losses for individual activities and pair-wise activity combinations for R = 100 rounds. After that, we use these results to estimate test losses of higher-order combinations following (Standley et al., (0200.. We then compute the actual test losses for the optimal activity splits that have the lowest test losses by training them from scratch. For example, for activity set sdnkt, we compute s, d, n, k, t and ten pair-wise activity combinations. Then, these results to estimate test losses of higher-order combinations.
TAG (Fifty et al. 202I) first computes all-in-one training for = 100 rounds to obtain the pair-wise affinities. Then, it
network selection algorithm to group these activities. After that, train each group of activities from scratch for R = 100 rounds to obtain test losses. The best result is reported for overlapping activities. For example, (sd, dn, kt} is the best result of three splits of TAG on activity set sdnkt. Then, each split is trained from scratch to obtain test losses.
Computation Resources Experiments in this work take approximately 27.765 GPU hours of NVIDIA Tesla V100 GPU for training. We conduct three independent runs of experiments for the majority of empirical studies. In each run, activity set sdnkt takes around 2,330 GPU hours, erckt takes around 3,280 GPU hours, and sdnkterca takes around 3,645 GPU hours. These include experiments of compared methods and ablation studies, whereas these do not include the GPU hours for validation and testing. It takes around the same GPU hours as training when we validate the model after each training round.
C ADDITIONAL EXPERIMENTAL EVALUATION
This section provides experimental results, including comprehensive results of performance evaluation and additional ablation studies.
py

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison of test loss, energy consumption, and carbon footprint on activity set sdnkt.
Mothod
|Splits EEneryy (KWh)
CO2cq (g)
Total Loss
k
One by one
8.4 + 0.1
2465 = 00603 + 0.030 + 0.005
0.261
0.023/0.10
0.001
0.107 = 0.003 0.043 = 0.002
Ale-in-ome
3.7 1: 0.1
1086 1 28 0.677 0.018 0.087
0.002 0.246
0.110
0.001
0.126 1 0.019 0.083 : 0.008
GradNorm
4.1
0.4
1200
0.691
0.001
002251
0.012
0.003
0.118
0.007
0.093 : 0.019
HOA
31.0
9125 140|0.611 0.029|0.091
1:011
0.002]0.135
0.000
0.003
0,074 = 0.023
TAG
2
9.8 4: 0.3
2876 1: 88 0.624
0.015
0.004
0.005 1: 0.001 0.110 +: 0.007 0.085 = 0.006
MuFL
4.9 0.3
1431 = 0.577 00000000000000000000000000000000
0.006
0.002 0.102
0.003 0.052 = 0.003
HOA
0.5
9125 1: 140|0.598 00200000000001110100000000000100 1:
0.007| 1: 0.008 0.107 1: 0.003 0.043 1 0.002
TAG
11.3
3313
0.233
0.002 0.2 0.013 0.1 0.008 0.055 0.008
MuFL
5.4 0.3
1589 = 94 00.555 = 0.00000000000000220000000000.0.0
0.006
= 0.002 0.095 = 0.003 00042 = 0.003
HOA
31.0 0.5
9125 + 140|0.597 0.0000000004440000000....0000...
0.002 115 = 0.014 0.107 = 0.003 0.043 = 0.002
TAG
13.7 0.3
4016 1 0.603
0.088
0.005
0.233 1: 0.002
0. 0.1 1 0.013 0.110 1 0.008 0.055 1 0.008
MuFl.
0.3
1969
0..448
0:001 0.070 0.002 0.230 0.008
0. 111 0.000 0.095 0.007 0.042 0.001
Table 4: Comparison of test loss, energy consumption, and carbon footprint on activity set erckt.
Method
|Splits|Energy (khh)
CO2ey (g)
Total Luss
One by one
11.1 + 22
3277 + 660 1,055 0,144
0.000
0.329
0.006 0.107 - 0.003 0.043 = 0.002
All-in-one
5.0 1: 0.3
1478 + 1.130 0.022
0,146
0.001 0.379
0.399 1 0.002 0.110 : 0.003 0,079 + 0.013
GradNorm
5.0
1: 0.2
1462 1 70
1.154 1 0.055
0.147 1: 0.002 0.381
0.015 0.394 1 0.001
0.149 1: 0.062 0.082 1 0.005
HOA
2
38.3 0.3
11265 + 86
1.082 :: 0.032/0.149 1: 0.003
1: 0.025 1 0.002 0.109 1 0.002 0.064 = 0.022
14.0 0.9
1: 279
1.095 0.033 0, 0.002 0.379
0.013 0.393 0.000
1:
: 0.015
MuFL
2
6.7
- 0.2
1957 = 53
1.039 = 0.000240.143333
: 0.001
0.344
= 0.0.014400333. = 0.001
0.104
0.006 0.056 0.007
HOA
38.3
1: 0.2
11265
53
1.062
0.024|0.149
0.001
(0.365
0.014
0.001 0.109 1: 0.006 0.046 1: 0.007
TAG
14.4
4 0.6
4242 + 170
1.091 + 0.00440.1447 :: 0.002 0.388
0.014 396 + 0.002 0. 109 = 0.009 0.050 = 0.011
MuFL
7.2 1: 0.2
2108
0.011 1.015
0.000
0.005 0.005
0.001 0.102 0.008
0.009
HOA
38.3 0.3
11265
86
1,053
0.03440.1448
0.002
0.369
0.028/0.386 + 0.006 0.105 + 0.001 0.045 + 0.003
TAG
17.4 1 0.5
5114 1: 159
1.087
0.028
0,147
0.002
0.384 1 0.011
0.39 1 0.002 0.109 0.009 0.050 1. 0.011
MuFT.
7.6 0.0
2229 =
1.02 - 0.014
0.143 + 0.000
0.336 1: 0.005
0. 383 + 0.001 0.094 + 0 0.009 0.046 = 0.004
C.1 PERFORMANCE EVALUATION
Table 3 and 5 provide comprehensive comparison of dif- ferent methods on test loss and energy consumption on activity sets scnkt and sdnkterca respectively. They complement the results in Figure 2 Besides, Table and
1.16
X GodtNon
2 Split Splits Splits
Allinoone
1.12-
Test Loss
1.10
FacProx
IAG
TMG
HOA Ø
1.00
HOA
Figure IT compares these methods on activity set e erc.t. The results on e erckt is similar results on the other ac- tivity sets; method achieves the best performance with around 40% less energy consumption than the one-by-one method and with slightly more energy consumption than all-in-one methods.
1.12-
Ours
1.00
Energy (kWh)
Figure 11: Compare test loss and energy consumption on activity set erckt.
Additionally. Table and also provide carbon foot-
prints (CO2eg) of differett methods. The carbon footprints are estimated using Carbontracker (Anthony al. 2020 10 Our method reduces around 40% on carbon footprints these three activity sets compared with one-by-one training; it reduces 15266C02ee or equivalent to traveling 12.68km by car on sankte erca. The reduction is even more significant when compared with TAG and HOA. Although we run experiments using Tesla V100 GPU, the relative results of energy and carbon footprint among different methods should be representative of the scenarios of edge devices.
C.2 ADDITIONAL ANALYSIS AND ABLATION STUDIES
This section presents additional analysis of MuFL and provides additional ablation studies.
Carbbnn intensity of a training varies over geographical regions according to (Anthony et al. 2020). We use the national level (the United Kingdom as the default setting of the tool) of carbon intensity for fair comparison across different methods. These carbon fontprints serve a proxy for evaluation of the actual earbon emissions.
18

---

Under review as a conference paper at ICLR 2023
Table 5: Comparison of test loss, energy consumption, and carbon footprint on sdnkt erca.
Method Splins
Energy
[C022e (g)
Total Loss
One by ond
150000000000000000
0551000
00000000000000000000000000000000
00:00000000
All-in-nee GraiNorm
4.220.2
14352200 1100:00011444000000000 00
100 0.00
144
1:12000 0
000000000
1.3
14:00.00
0.3.10.000003
0.391000
1:00000000
TAG MuFL
2 122.A:1hhh0100000000000000000110
149
014
+0001
6.0:5:0.1 (986:) 108
0.021 0.08 10000000000000000000000000000000
0120001000000000000
10001
TAG
MaFT.
(9852
TAG
15.8:22,44699: 1.4 4.0000000000007700000000000 1:t0::00000000.000333.000003
MaFL
7.5220.3 2201:194 00000000000000000000000000000000
MoFL.
|8.3220.4 [43320044499::00000 1139:44 00000100000000000000000000000000
88 4.76
2,108
5.81
2.08
2.29
4.74
2.20
Voldddeen
3.61
1.47
1.48
2.54
0
60 Training Round
23
90
100
20
2
90
10c
0
23
60
90
100
Training Round
Training Round
(a) Activity set sdnkt
(b) Activity set erckt
(c) Activity set sdnkterca
Figure 12: Changes of validation loss over the course of training activity sets: (a) sdnkt, (b) erckt, and (c) sdnkterca. Validation loss converges as training proceeds.
Table 6:
Activity splitting results of TAG (Fifty al. 2021 and MuFL on activity sets sdnkt, erckt, and sdnkterce. Activities of each split is separated by comma.
Method
Activity Set
Two Splits
Three Splits
Four Splits
Five Splits
TAG
sdnkt sdnkt
sdn,kt sdn,kt
sd,dn.kt sdn,k,t
sd,sdn,dn,kt sd,nk,tt er,,t,rc,it
MuFL
s,d.n.k.t
TAG
erckt
er,rekt er,ckt er.ckkt er,c,k,t e.r.c,k.t
er,kt,rc
MuFL
erckt
TAG
sdnkterca sdnkterca
sd sdtere,dr,kta se,dr,ne,kta snkteac,dr
MuFL
snec,dr,kta sn,dr,ka,ete sn,dr,ka,e,tc
Changes of Vadiation Loss Figure 12 presents validation losses over the course of all-in-one training of three training activity sets sönkt, erckt, and sdnkterca. It shows that validation losses converge as training proceeds.
Splitting Results of Various Methods We provide results of activity splitting of TAG (Fifty et al. 2021) and MuFL in Table 6 For hicrarchical splitting, they further split into three splits from the results of two splits. In particular, the results of hierarchical splitting of e erckt and sdnkte are the same as their three splits. The hierarchical splitting result sdnkt is from (sdn, kt to {sd, k the hierarchical splitting further divides the split with more training activities (sdn).
Besides, Table presents the splitting results of the optimal and worst splits. They are not identical due to variances multiple runs of experiments. We report the mean and standard deviation of test losses of the optimal splits and the worst splits in Table The large variances of the optimal and worst splits suggest the instability of splitting by measuring the performances of training from scratch in the FL settings and show the advantage of our methods in obtaining stable splits.
Dataset Size and Performance The dataset size of activity set sdnkt is around 315GB in our experiments, compared to 2.4TB dataset used in experiments of TAG (Fifty et al. 2021). The test loss of ours (0.512 in Table 8), however, is better than the optimal one in TAG (Fifty et al. 2021) (0.5246). This thee----thh-eeeelppeee comparison indicates the potential to extend our approaches to multi-task learning. Besides, it could also suggest that our data size is sufficient for evaluation.
19

---

Under review as a conference paper at ICLR 2023
Table Results of the optimal and worst splits in three runs of experiments. They are not identical due to variances in three runs of experiments.
Activity Set
Splits
Optimal Splits
Worst Splits
2
dk,snt
sn,dkt k,t,sdn
nt,sdk d,sn,kt
st,dnk d,st,nk
st.dnk d,st,nk
st,dnk
sdnkt
t,sn,dk
s,dt,nk
r,eckt
t.erck r,t,eck
et.cck r,ec,kt
rk,ect c,e.rk
ek,rct e,rckt e,k,rct
erckt
r,eckkt
e,rt,ck
0.16
2.56
0.60
Affinty
6.12
Minity
1.522
Affinit
0.445
0.08
1.38
0.30
0.00
1.64
00.15
0.00)
0.00
0.00
Training Round
Training Round
Training Round
(a) Affinities to activity 8
(b) Affinities activity
(c) Affinitics activity
0.12
1116
wwmm
0.16
Affiiity
0.009
Affinty
0.12
0.12
0.00
1L06
0.08
0.03
0.01
20.44
0.00
0.00
0.00
60
100
60
100
20
211
60
100
Training Round
Training Round
Training Round
(d) Affinities to activity k
(c) Affinities to activity
(f) Affinities activity
1.36
0.06
Affiety
0.34
0.08
0.02
Affeei
0.10
0.20
0.16
00005
0.00
0.00
20
60
30
100
20
40
60
50
100
0
20
40
60
80
100
Training Round
Training Round
Training Round
(g) Affinities to activity=
(h) Affinities to activity C
Affinities to activity
Figure 13: Changes of affinity scores of one activity to the other over the course of training on activity set sdnkterca. The trends of affinities emerge at the early stage of training.
Impact of Affinity Computation Frequency f The frequency of computing affinities in Equation 2 determines the amount of extra needed computation. We use f f 5 and compute affinities for the first ten rounds for all experiments because the trend of affinities emerges in the early stage of training in Figure 13 It would increase the computation of all-in-one training by around 2%, which is already factored into the energy consumption computation previous experiments. The results Table 3 and 5 show that MuFL is effective with this setting and the amount of computation is acceptable.
Impact of Local Epoch Figure 14a show the impact of local epoch E on activity sets sdnkt, erckt, and sdnkterca. They complement results of activity set adnkt in Figure [a Larger E could lead to better performance with fixed R = 100. It is especially effective when increasing 1 to E = 2, but further increasing E could degrade the performance. indicates that simply increasing computation has limited capability to improve performance.
Impact of The Number of Selected Clients Figure 14b compares the performance of differ- ent numbers of selected clients K = {2, 4, 6, 8, 16} on three activity sets sünkt, erckt, and sdnkterca. It complements results in Figure 5b The results on three activity sets are similar; increasing K reduces losses, but the marginal benefit decreases as K increases.
The majority of experiments in this study are conducted with K 4. We next analyze the impact of K in MuFL with results of two splits activity set sdnkt in Table The results indicate that MuFL is still effective with K = 8, which outperforms = 4 and all-in-one training.
Standalone Training Standalone training refers to training using data of each client independently. Figure [5a shows the test loss distribution of thirty-two clients used in experiments. The client ID corresponds to the dataset size distribution in Figure These results suggest that larger data sizes of
20

---

Under review as a conference paper at ICLR 2023
1.6
1.6
[ Loss
Test Loss
1.2
1.2
0.8
0.8
0.4
0.4
sdnkt
erckt
sdnkterca
sdnkt
erckt
sdnkterca
E
(a) Impact of E
(b) Impact
Figure 14: Analysis of the impact of local epoch E and impact of the number of selected clients K. Larger E (with fixed R = 100) and K requires higher computation. They could reduce losses, but the marginal benefit decreases computation increases.
Table Comparison of test loss using different numbers of selected clients K. MuFL achieves even better performance on K 8.
K
Total Loss
d
k
All-in-one
4
0.677
0.087
0.246
0.136
0.126
0.083
All-in-one
0.618
0.076
0.227
0.130
0.109
0.077
MuFL (two splits)
4
0.578
0.069
0.231
0.124
0.102
0.052
MuFL. (two splits)
8
0.512
0.060
0.202
0.117
0.083
0.048
2.6
2.2
Methods
Test Loss
Standalone
1.842 0.248
0
1.8
All-in-one
0.677 + 0.018
1.4
MuFL 0.548 = 0.001
30
Client ID
(a) Test loss distribution of standalone training
(b) Test loss comparison
Figure 15: Performance (test loss) standalone training that conducts training using data in each client independently: (a) shows the test loss distribution of these thirty-two clients. (b) compares test losses of standalone training and FL methods. We run the experiments on activity set sdnkt.
clients may lead higher performance. Figure 15b compares losses standalone training and federated learning methods. Either all-in-one or our MuFL greatly outperforms standalone training. It suggests the significance of federated learning when data are not sharable among clients.
21

---

Under review as a conference paper at ICLR 2023
D ALGORITHM
Algorithm Our Proposed Smart Multi-tenant FL System (MuFL)
1: Input: training activities A = {ai, 02...., Ohp J. set of available clients C, number of selected clients K, local epoch E, aggregation weight of client k Pk, total training rounds R. all-in-one training rounds Ro. the number of splits frequency of computing affinities f, batch size B {w1. w2,,,w2,....... .,,,,,
2: Output: models W
3:
4: ServtrExecution:
5: Consolidate A into do with multi-tast task model 10 {0a) 10,0,/ E A} Act. consolidation
6: Initialize vº, i.c. initialize wi = {0.] U {0a,} } for E {1,2..... 0, 1, l,... Rg - do
7: for each round
8: C + (Randomly select clients from C)
9:
for client k C' in parallel do w ,a,-aa Client(v", A. f)
10:
11: end for
12: ++1
Σ P.EeT. REC"
13:
-a f Σ
Srikk Di -00
kEe
end for
15:
Compute the values of Sx -a Va, E A, using Equation Compute a disjoint partition set activities for splits {A,li 1} that maximizes St, using affinity scores S Sa,+ay" Va; E A and Vaj= E A
Compute affinity scores
16:
Activity splitting Schedule to train sequentially
17:
for each element j E do
18:
Initialize v, = [0: U {0o, a E A, with parameters of To for each round
19:
0, 1,... Ro 1 do
20:
C + (Random select K from C) for client l C" in parallel do
21:
with :- -.--Clientuu, Aj. 0)
22:
23:
end for
24: r+
Σ PRut kEC
25: end for
26: end for
27: Reconstruct W =
from (uyli E 7) by matching training activities
28: Return W
29:
Client (v, A, f):
B 31: T if
# else
32: for local epoch e = I.... do
33:
Update model parameters with respect to training activities
34: for each time-step 1, 2, ...,T (every f batches) do
35:
Va, and Va, A, compute affinities of St Saytay using Equation 2
36: end for
37: end for
E
S
St aa
Yas E and Vay e A
ET
39: Return V/ Sau
22


Reviews:
[{"review_id": "zP3RxMrbv0", "reviewer": "Reviewer_NZnW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission is considering if multiple federated learning tasks on a single device can be coordinated and consolidated to ease resource constrain and energy consumption. A multi-tenant FL system called MuFL is proposed that 1) Initially merges \"similar\" training activities into one activity with a multi-task architecture; 2) After some rounds of training, splits back the activities it into groups to have a better synergy. On two dataset empirical studies are carried out which show marginal benefits.", "summary_of_the_review": "Even though important problem setup, the submission has limited technical innovation and marginal empirical improvements, thereby unfortunately warranting a low overall review score.", "strength_and_weaknesses": "**Strength**\n- Simplicity of proposed approach\n- Easy to follow the paper\n\n**Weakness**\n\n- Limited empirical evaluation and significance of the improvements is not clear (both in terms of test loss and energy)\n- The source of minor improvement in test loss is also not clear: Maybe the improvements are just originating from multi-task learning due to sharing between multiple activities\n\n- Limited applicability of the setting: It is assumed same server or same service provider is responsible for all the activities in multi-tenant FL. What if server for different activities are different? A much more practical and common scenario would be when the service provider of the keyboard app and camera app are different, i.e. keyboard server can be different from camera server. How would the proposed MuFL be beneficial in that case?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "w1ToDPe6Yqf", "reviewer": "Reviewer_CNLB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to train multiple simultaneous activities on decentralized edge devices with budget constraints. Most prevailing approaches aim to do this by training tasks one at a time. However, the paper proposes a solution of splitting the tasks into groups and learning jointly. This enables sharing of information as well as an improved efficiency in terms of energy consumption.\n\n", "summary_of_the_review": "Although the results are interesting, it is difficult to truly understand the paper without the complete training regime. The lack of the loss function used is a deterrent, in my view. Besides that, I feel the approach is limited as it can only be used in Scenario 1 (as per Section 3.2).  ", "strength_and_weaknesses": "**Strengths**\n* The paper proposes a new domain/problem which could be interesting to AI researchers.\n* The paper has ample experiments that there are no issues with pareto-optimality, etc. as usually faced in meta-learning and multi-task learning problems. \n\n**Weaknesses**\n* I believe the implementation could be discussed in more detail. The reader is left wondering what the loss was. How come there has been no non-pareto-optimal solutions when training with different splits? \n* Although the problem is interesting, I am afraid the limitation of the approach is very closely tied to privacy. For example, suppose a device has training activities related to two different companies/entities. It might not be possible to force the two networks to be same while training, or even train with the same data due to data leakage, privacy concerns, etc. This highly weakens the potential impact of the work. If I have misunderstood this part, I hope the authors could correct me here.\n* Although the authors talk about 4 different scenarios in Section 3.2, I believe all experiments could be classified under scenario 1, and the approach has not been shown to be applicable to all 4 scenarios.\n* In section 5.5, the authors show that increasing the number of epochs seems to harm the overall test loss. I would assume this is a classical scenario of overfitting, but I hope the authors could explain this. Especially since the exact formulation of the loss used for training is not detailed, this is hard to guess.\n* This is more of a question: In Table 3, why is the loss of one-by-one higher than the loss of all-in-one for the task 'd'? This intuitively should not be the case since the all-in-one will simplicity come with more noise.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "raFnE5bxHu", "reviewer": "Reviewer_hdrV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel setting, which is the multi-tenant federated learning especially in edge computing scenario. Specifically, multi-tenant learning aims to optimize/train models which are used for multiple tasks or optimization goals. Federated learning aims to explore subset of the training data for computational resources and private considerations. There is no well explorations which consider both of these requirements, while it is a practical problem in general applications especially in recent years. To this end, this paper proposed 1) a novel setting of multi-tenant federated learning, 2) a vanilla solution of this task, and 3) specifically designed solutions for this task. Extensive experiments are conducted to show the effectiveness and efficiency of the proposed model compared with current general solutions.", "summary_of_the_review": "This paper proposed a novel setting of multi-tenant federated learning and provides a solution for this problem.", "strength_and_weaknesses": "Strength:\n\n1 This paper proposed a novel setting which is multi-tenant federated learning. It aims to solve the multi-tenant problem in federated learning setting. Four different scenarios are introduced.\n\n2 One vanilla solution and the specifically proposed pipeline are described.\n\n3 Extensive experiments demonstrate the effectiveness and efficiency of the proposed model compared with baselines.\n\nWeaknesses:\n\n1 Due to the novel setting of the problem, the baselines are also the vanilla approach. The significance of the performance improvement.\n\n2 It is not common (general requirements) in edge/device sides to obtain all related and multiple data formats in the same time and to conduct the multi-tenant training. More potential applications/samples are recommended to discuss in the introduction section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7h2VXSYxu2n", "reviewer": "Reviewer_icP7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is looking into the multi-tenant federated learning problem, which is considering training multiple federated learning tasks at the same time. The goal of this work is to minimize power consumption while maximizing the performance of training results. This work proposes using a multiple-task share-model technique followed by a splitting method to minimize training loss while reducing power consumption.", "summary_of_the_review": "Please see above. Overall, I think this paper needs more work to improve its novelty and contribution.", "strength_and_weaknesses": "**Strength**:\n- The general idea proposed by this work is relatively easy to follow.\n- The problem of multi-tenant federated learning is interesting. The paper provides an easy-to-understand summary of different types of multi-tenant federated learning.\n\n**Weakness and questions**:\n- This contribution of this work seems somewhat limited. The challenge of the problem is not too clear, the performance gain seems be coming from multi-task training which is from existing literature. It seems to be a directly application.\n- For the proposed method, the hyper parameters (e.g. R0) seems to be hard to select, and it is more challenging as you have more different types of tasks which might be unknown in advanced. To support multi-tenant, it would be more preferrable to have a method that can automatically selects these hyper parameters.\n- It's unclear how the affinity score is obtained. If the training are done in groups (splits) or all-in-one, then don't they share the same model, how do you measure the affinity score between them? Do you need to do additional computation to get that pairwise affinity score. This splitting decision seems to be exponential related to the number of tasks because you need do pairwise comparison and group selection. It's unclear how this is done.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject