Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: ## The problem
The paper introduces an asynchronous  aggregation scheme to address the straggler problem and also taking in the effect of non-iid. The paper focuses on "heavyweight" learning task such as video action recognition that are out of reach for mobile devices processing power. Overall, the paper aims to address the problem of how to best aggregate the updates sent by all clients in order to maximize information
learned while minimizing any adverse effect from slow updates. 
## The solution
Scale client update based on its staleness and the client's local training loss. The aggregation scheme gives more importance to the clients with low training accuracy to push to global model to learn from clients with low accuracy. 



strength_and_weaknesses: ## Strength
1. The paper is easy to read and follow. The problem and motivation is clearly stated in the introduction. 
2. The aggregation scheme is simple and can be easily incorporated without making any new assumptions. 
3. The takeaways section helps capture the high level points of the paper. 

## Weakness
1. Equation 2 is hard to follow. For example, what is $w_{new,t}^i$ ?
2. The video action recognition experiments were done on a very small number of devices (less than 12). FedBuff is design for cross-device setting with very high concurrency and K to be a small fraction of concurrency. 
3. In large scale experiments (Figure 5), the difference between Kuiper vs Oort and FedBuff seems negligible. Did the author repeat the runs for multiple trials and report the average?
clarity,_quality,_novelty_and_reproducibility: 1. I would suggest the authors to rewrite equation in a more understandable manner. From my understanding, there is a timeout threshold $T_0$ which determines $w^{c^t}_{new,t} = 1 - \text{acctrain}^i_t \space \text{if} \space t <T_0\ $  and 1 otherwise. 

2. It seems like the difference between KUIPER and FedBuff is in equation 4 with the additional term $ \beta^{c_t}_t w^{c_t}_{new,t} $, is that correct?

3. Why is heavyweight ML task an important problem setting? In FL with resource constrained devices, the learning problem is small and lightweight. I would love to hear from the authors on why heavyweight ML task should be done in a federated setting and not at the server?

4. The point about KD is confusing, why did the authors chose KD instead of simpler methods such as pre-training? Why is studying the impact of KD relevent to this work?
summary_of_the_review: Overall, I believe this paper is a simple extension to FedBuff. However, I think the improvement is incremental and the problem of FL on heavyweight ML task seems like a niche problem space. 

## Post-rebuttal update
The authors answered my concerns about scalability and clarified the difference between KUIPER and FedBuff. However, I think the paper can benefit from throughout revision to make it clear what the contributions and focuses of the paper are. Secondly, the paper lacks details about the how the client execution time, staleness and how stragglers are simulated. I appreciate that the authors did clarify these point in their comments. 

I will stand by my initial rating of below the acceptance threshold. I think this is good contribution to the community once the weaknesses are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Kuiper is a federated learning system designed to produce accurate global models in the presence of non-IID client data.   In particular Kuiper addresses the straggling clients with an asynchronous, buffered aggregation technique.   Kuiper proceeds in rounds, starting a round when K client updates arrive.  It then adjusts the global model based on client data set size, update latency, and local training error.  The work provides a convergence analysis and experiments show that the system provides comparable or better prediction performance against recent work on different video action recognition benchmarks.    
strength_and_weaknesses: Positives
+ Provides a delta on top of existing work in sync/async buffered approaches for dealing with stragglers. 
+ Extensive experiments on a challenging modeling task across multiple datasets against competing systems 
+ Experiments showed training/time benefits, and explored system design (scale, tuning variables)  

Negatives
- The description of the algorithm, particularly the use/measure of time, is not clear
- While the system compares to Oort / FedBuff, the differences aren't clear and spread across the paper 
- The writing quality and organization makes understanding the contributions and system challenging. 
clarity,_quality,_novelty_and_reproducibility: Thanks for the submission -- this work appears to make contributions in exploring how best to incorporate non-IID updates in an asynchronous fashion.   There were some areas where I had additional questions / confusion. 

Figure 1 is greatly appreciated, but I was unable to rectify that figure and the description of the update policy.   In particular, Section 3 don't appear to define what t or \tau (or \tau_{i}) are.  In particular, are they "indices" of the t-th round?  Or are they actual measurements of latency?   If you look at the figure, each arrow representing t-tau appears to be the time between updates for each client.  That means t and tau are in terms of time.    In the figure, is t from t_2 or t_3?  Then we look at eq 2.  T_0 is a threshold, so assume it's a point in time.  But here t is the t-th iteration?  Why is t the same across clients?   Equation 3 raises similar issues.  s() makes sense if s and tau^{ct} are update index, but not if they are clock time (t continues to grow while the average delay does not).      

The paper does a nice job making sure that relevant work is cited.  However, it was difficult to figure out exactly how Kuiper is different from its most closely related systems (Oort/FedBuff).   One confusion arises where Oort is described in the Async FL systems (Sec 2 P2).   But then is described as synchronous in sec 5 burst size paragraph.   There is language describing how Oort also measures client utility, but never it is precise enough to compare to Kuiper's approach.   Sec 2 P2 says Oort "waits for a fixed K to 'synchronize'" -- sounds just like Kuiper.  Then sec 5 baseline comparison says that Oort "gives the same weight to all client updates" -- not sure what that means exactly.  
Then in the burst size paragraph it says Oort "waits for the K chosen clients in each epoch."   The paper should precisely describe the competing system in one place to make it utterly transparent how Kuiper is different. 

Sometimes the work seems confused about its ultimate contribution.  Is it for doing action recognition at edge devices or is it about handling stragglers when doing FL?  This happens in Section 5 where the first question that comes to mind is whether that task can run on edge nodes.  I had no idea that was the *main* point of the paper.  The last question is whether it handles stragglers.  It's also the very smallest graphs in the paper.  You should refer to appendix for the alpha/beta question. 


Lesser items:

* Including KD in design was confusing.  The paper reads as if KD creates the initial scaled-down edge models, and then Kuiper (and competing systems) are used to "fine tune" the client models.   In this case, what does KD have to do at all with the system design?  It seems like it would only really impact your implementation and experiments.  One doesn't need KD to use Kuiper, right? 

* Sec 2 P2 and P3 also have nearly the same content.   P2 ends with issues with non-IID then P3 starts by saying the same thing. 

* Sec1 page 2, footnote 3, you mean "Kuper's superiority" not "our superiority" right? :) 

* The paper treats the prediction task as more than just a good, complex model to train.   The "Takeaways" section says this is the first time we can run action recognition on edge devices, but we could have just installed the centrally trained model.  In addition, while I'd agree that the 9-10% gains are meaningful, it's not because the current central accuracy is 47%.  47% is still awful - FL techniques won't improve upon centrally trained model performance.    

* The paper describes averaging and merging client gradients but eqs 2,3,4 all use weights w.  I don't think it invalidates the work, but it's not consistent. 

* Sec 2 P3 says highly-skewed distributions happen in mobile computing environments.   What's the basis for that statement?  Are we talking about mobile computing or edge computing here (experiments seemed to be edge devices)? 

summary_of_the_review: Moderate contribution in asynchronous FL.   Strong evaluation / extensive results on a non-trivial problem across datasets, against competing algorithms, including exploring various tuning variables.   The writing and paper focus makes understanding details of the technical contribution difficult, as well as placing it in the context of the systems to which they compare.  

[Post Author Response].  I'd like to thank the author's for explaining the questions my review raised.  The work unifyies techniques from other systems in a non-trivial manner, which is an important contribution for the community.   The authors address many of the presentation problems, and the system performance shows clear global model and training time improvements.    In view of this I'm adjusting the empirical significance and recommendation to a 6.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors tackle a practical problem of federated learning where asynchronous communication is frequent due to the computational heterogeneity. The authors propose a method considering taleness of local updates when performing model aggregation. They demonstrate the effectiveness of their method.

strength_and_weaknesses: **Strength**
- They tackle the practical federated learning where asynchronous communication is frequent due to the computational heterogeneity.
- The results are impressive.

**Weaknesses**
- The critical limitation of this work is the lack of significance. The proposed method for calculating staleness seems a rule-based algorithm which is hard to consider as significant. I think it could be a better way to devise a neural model learning the staleness and importance of asynchronously updated local models (neural aggregator, etc). 
- Error bars are missing and thus it is hard to discuss the marginal case i.e. Fig 4 (b) 
- Minor:
    - Paper organization need to be polished (inbetween spaces are too tight)

clarity,_quality,_novelty_and_reproducibility: There are some rooms for improving the quality, clarity and originality of the proposed method, as mentioned above. 

summary_of_the_review: I enjoyed reading the paper, but several improvements are required.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approach to asynchronous federated learning. The focus is on FL or video action recognition, and the paper proposes a new approach aggregating client responses in async FL. Experiment results suggest the proposed approach outperforms previous approaches FedAsync, FedBuff, and Oort.

One major weakness of the initial submission was the clarity of presentation in the paper. All reviewers raised concerns about this and difficulty of understanding the difference between the proposed approach and previous work, as well as understanding what constitutes the main contribution of this paper. The author responses helped to address some of these concerns, especially for one reviewer who raised their score. However the issues remained for other reviewers. Although the responses helped with clarifications, it doesn't appear that any revised version of the paper was updated during the rebuttal period, and it would have been very helpful to see concretely how the paper would be changed to address these concerns.

Other concerns were raised about missing details of the experimental setup. These were somewhat clarified in the rebuttal, but again it would have helped to see a revised manuscript implementing all of the changes.
justification_for_why_not_higher_score: Given the current reviews and discussion, my recommendation is to reject this paper. It wasn't discussed, but my sense is that the discussion wouldn't have changed anything.

There is a possible argument for raising this to Accept (poster). That would be to (a) discount the input of Review t9vz, which is both the lowest score and also somewhat lower confidence, and (b) to put a lot of faith in the authors to accurately implement all of the changes and clarifications that came about based on feedback from the reviewers.

However, the number of changes required to improve the presentation to be acceptable is substantial enough that they should be reviewed again before the paper is accepted. If the authors had uploaded a revision during the rebuttal period, then the reviewers and AC could have seen the changes and made a more confident call. Since no revision was uploaded (as far as I can tell, the most recent version was uploaded in October), this isn't an option.
Decision: Reject