Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method to train multiple simultaneous activities on decentralized edge devices with budget constraints. Most prevailing approaches aim to do this by training tasks one at a time. However, the paper proposes a solution of splitting the tasks into groups and learning jointly. This enables sharing of information as well as an improved efficiency in terms of energy consumption.


strength_and_weaknesses: **Strengths**
* The paper proposes a new domain/problem which could be interesting to AI researchers.
* The paper has ample experiments that there are no issues with pareto-optimality, etc. as usually faced in meta-learning and multi-task learning problems. 

**Weaknesses**
* I believe the implementation could be discussed in more detail. The reader is left wondering what the loss was. How come there has been no non-pareto-optimal solutions when training with different splits? 
* Although the problem is interesting, I am afraid the limitation of the approach is very closely tied to privacy. For example, suppose a device has training activities related to two different companies/entities. It might not be possible to force the two networks to be same while training, or even train with the same data due to data leakage, privacy concerns, etc. This highly weakens the potential impact of the work. If I have misunderstood this part, I hope the authors could correct me here.
* Although the authors talk about 4 different scenarios in Section 3.2, I believe all experiments could be classified under scenario 1, and the approach has not been shown to be applicable to all 4 scenarios.
* In section 5.5, the authors show that increasing the number of epochs seems to harm the overall test loss. I would assume this is a classical scenario of overfitting, but I hope the authors could explain this. Especially since the exact formulation of the loss used for training is not detailed, this is hard to guess.
* This is more of a question: In Table 3, why is the loss of one-by-one higher than the loss of all-in-one for the task 'd'? This intuitively should not be the case since the all-in-one will simplicity come with more noise.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is well-written and reader-friendly. The tables do look a bit off, but that could be easily fixed, and the paper is readable. 

**Reproducibility**

I believe the work lacks significantly in reproducibility. Although the basic hyperparameters such as number of splits, local epochs have been provided. The paper does not go into the network structure used, the loss functions, the learning rate and other crucial details that are required to replicate the work.

**Novelty**
The authors try to offer an approach to train multiple activities on a resource-contrained device.

summary_of_the_review: Although the results are interesting, it is difficult to truly understand the paper without the complete training regime. The lack of the loss function used is a deterrent, in my view. Besides that, I feel the approach is limited as it can only be used in Scenario 1 (as per Section 3.2).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission is considering if multiple federated learning tasks on a single device can be coordinated and consolidated to ease resource constrain and energy consumption. A multi-tenant FL system called MuFL is proposed that 1) Initially merges "similar" training activities into one activity with a multi-task architecture; 2) After some rounds of training, splits back the activities it into groups to have a better synergy. On two dataset empirical studies are carried out which show marginal benefits.
strength_and_weaknesses: **Strength**
- Simplicity of proposed approach
- Easy to follow the paper

**Weakness**

- Limited empirical evaluation and significance of the improvements is not clear (both in terms of test loss and energy)
- The source of minor improvement in test loss is also not clear: Maybe the improvements are just originating from multi-task learning due to sharing between multiple activities

- Limited applicability of the setting: It is assumed same server or same service provider is responsible for all the activities in multi-tenant FL. What if server for different activities are different? A much more practical and common scenario would be when the service provider of the keyboard app and camera app are different, i.e. keyboard server can be different from camera server. How would the proposed MuFL be beneficial in that case?
clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is overall clear. Minor: $\mathcal{C}$ not defined before first used in Sec 3.2
- Novelty: the novelty seems limited 
- Reproducibility: All hyperparameters and configuration is provided in appendix B
summary_of_the_review: Even though important problem setup, the submission has limited technical innovation and marginal empirical improvements, thereby unfortunately warranting a low overall review score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel setting, which is the multi-tenant federated learning especially in edge computing scenario. Specifically, multi-tenant learning aims to optimize/train models which are used for multiple tasks or optimization goals. Federated learning aims to explore subset of the training data for computational resources and private considerations. There is no well explorations which consider both of these requirements, while it is a practical problem in general applications especially in recent years. To this end, this paper proposed 1) a novel setting of multi-tenant federated learning, 2) a vanilla solution of this task, and 3) specifically designed solutions for this task. Extensive experiments are conducted to show the effectiveness and efficiency of the proposed model compared with current general solutions.
strength_and_weaknesses: Strength:

1 This paper proposed a novel setting which is multi-tenant federated learning. It aims to solve the multi-tenant problem in federated learning setting. Four different scenarios are introduced.

2 One vanilla solution and the specifically proposed pipeline are described.

3 Extensive experiments demonstrate the effectiveness and efficiency of the proposed model compared with baselines.

Weaknesses:

1 Due to the novel setting of the problem, the baselines are also the vanilla approach. The significance of the performance improvement.

2 It is not common (general requirements) in edge/device sides to obtain all related and multiple data formats in the same time and to conduct the multi-tenant training. More potential applications/samples are recommended to discuss in the introduction section.

clarity,_quality,_novelty_and_reproducibility: This paper has clear motivation of the novel setting. The proposed method is clearly introduced. This paper has comprehensive experiments.
summary_of_the_review: This paper proposed a novel setting of multi-tenant federated learning and provides a solution for this problem.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work is looking into the multi-tenant federated learning problem, which is considering training multiple federated learning tasks at the same time. The goal of this work is to minimize power consumption while maximizing the performance of training results. This work proposes using a multiple-task share-model technique followed by a splitting method to minimize training loss while reducing power consumption.
strength_and_weaknesses: **Strength**:
- The general idea proposed by this work is relatively easy to follow.
- The problem of multi-tenant federated learning is interesting. The paper provides an easy-to-understand summary of different types of multi-tenant federated learning.

**Weakness and questions**:
- This contribution of this work seems somewhat limited. The challenge of the problem is not too clear, the performance gain seems be coming from multi-task training which is from existing literature. It seems to be a directly application.
- For the proposed method, the hyper parameters (e.g. R0) seems to be hard to select, and it is more challenging as you have more different types of tasks which might be unknown in advanced. To support multi-tenant, it would be more preferrable to have a method that can automatically selects these hyper parameters.
- It's unclear how the affinity score is obtained. If the training are done in groups (splits) or all-in-one, then don't they share the same model, how do you measure the affinity score between them? Do you need to do additional computation to get that pairwise affinity score. This splitting decision seems to be exponential related to the number of tasks because you need do pairwise comparison and group selection. It's unclear how this is done.

clarity,_quality,_novelty_and_reproducibility: *Clarity: the paper is overall clear
*Novelty: the novelty seems limited
*Reproducibility: no code is provided, so this is unknown
summary_of_the_review: Please see above. Overall, I think this paper needs more work to improve its novelty and contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper attempts to improve decentralized training on edge devices using federated learning, like reduce energy footprint. In this regards, authors propose to coordinate and consolidate multiple federated learning tasks on a single device. A multi-tenant FL system called MuFL is proposed that 1) Initially merges "similar" training activities into one activity with a multi-task architecture; 2) After some rounds of training, splits back the activities it into groups to have a better synergy. On two dataset empirical studies are carried out which show marginal benefits. We thank the authors and reviewers for actively engaging in discussion and improving the paper like adding loss function details. However, despite authors responses unfortunately none of the reviewers were convinced about the novelty and technical depth. There are unresolved concerns about how loss  between different tasks interact, and where does the improvement arise from: is it just multi-task learning? Also, the significance of the improvements is not evident. Furthermore, all reviewers are questioning the practical application of the proposed method, but that is of lower importance for an academic conference. 
justification_for_why_not_higher_score: Despite authors responses unfortunately none of the reviewers were convinced about the novelty and technical depth. There are unresolved concerns about how loss  between different tasks interact, and where does the improvement arise from: is it just multi-task learning? Also, the significance of the improvements is not evident. Furthermore, all reviewers are questioning the practical application of the proposed method, but that is of lower importance for an academic conference.
Decision: Reject