Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a mechanism called global batch gradient aggregation (GBA) on parameter server architectures that enables effective switching between synchronous and asynchronous training of deep learning-based recommendation systems. 
strengths_and_weaknesses: Strengths:

- The discovery of the phenomenon that directly switching between synchronous and asynchronous training of deep learning-based recommendation models leads to a significant drop in performances is interesting.  

Weaknesses:

- The explanation of the phenomenon is lack of technique depth. The insights presented in Section 3.2 are intuitive rather than analytical. It would make the paper stronger if there were some formal theoretical analysis on the drop in performance.  

- The performance boost is marginal; for example, in Figure 6, it is very difficult to understand the performance gain w.r.t. the proposed method. To be more concrete, it seems that, for the Criteo dataset, synchronous training achieves the optimal AUC much ealier than GBA. 

questions: Figure 6 is hard to track, would it be possible to list the final AUC of each model for each task in a table?
limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel algorithm to switch between two common distributed training modes (abbreviated as PS and AR in the paper) without sacrificing accuracy and performance while also not tuning hyperparameters. The paper identifies three key observations that govern performance and accuracy and based on these, proposes a global batch aggregation scheme that mimics the global gradients in synchronous training to switch between PS to AR without any loss in accuracy. The paper shows a convergence analysis of the algorithm and evaluates its performance on three different tasks showing it has better performance and accuracy overall.

strengths_and_weaknesses: Strengths:
- The paper proposes a novel algorithm to effectively switch between two training modes without tuning hyper parameters which is costly.
- he paper theoretically proves convergence.
- The proposed algorithm performs better than state-of-the-art recommender training schemes on three different tasks.

Weakness:
- The paper should provide details on when to initiate switching based on cluster status and ablation studies that show how the model performs for different cluster statuses. 

questions: - How does the performance vary with the number of workers? Since it relies on global batch aggregation, there may be some dependence on the number of workers?
limitations: The authors discussed the limitations of the work in the conclusion section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors of the paper "GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Models" propose a new training method for the recommender systems that enables switching between two different modes: synchronous and asynchronous training benefiting from both approaches and allowing for continuous learning maintaining the same accuracy and efficiency without hyperparameter tuning. Users of such a system can select a suitable training mode according to their judgment of cluster workload. 
strengths_and_weaknesses: Post Rebuttal comments: After the changes provided by the authors, I've updated my score to accept

------------------------------------------
Originality

The paper solves a problem that, though specific to recommender systems, can be very beneficial in a practical setting and valuable for all the industries attempting to train large recommendation systems under imperfect computing constraints. I would judge that the originality of the paper is high as, according to the authors, nobody has attempted to provide a hyperparameter tuning-free approach to the switching between synchronous and asynchronous training modes. 

Quality

The quality of this paper is also high - it provides a clear overview, and outlines several important insights and observations on the performance of the training modes that help to determine and address the gaps in the current work.  This provides clarity on the choices and explains the conducted experiments. 

Clarity

The paper is well-structured and well-written. I enjoyed reading it. The only problem that undermines the clarity is Section 5.2 and more specifically, Figure 6. I will outline my confusion in the Questions section below. In summary, it seems that the methods in the figure and their performance do not correspond to the description in the text. 

Significance

I find that this work might have a medium impact in addressing important challenges specific to the recommender systems' properties in the industrial setting. I believe that this impact might translate to better resource utilization and, therefore, cost savings.  
questions: I have only a couple of questions:
* In Section 5.2 you write that "We can see
261 that the low gradient staleness helps Hop-BS receive a better accuracy than BSP" (line 260-261). When I look at Figure 6 (a-c), I observe that HOP-BS performs the worst in the (a) setting dropping right after the beginning (Test day 14) while in other settings (b,c) it is not possible to distinguish any difference in AUC performance. BSP (dark blue) is performing clearly better. Next, you write that HOP-BW performs the worst, while I observe a relatively normal performance. I assume HOP-BS and HOP-BW got mixed up here. However, it doesn't explain that you say that GBA improves the AUC on (a) and (c) by at least 0.2% but I can't see this improvement in (a) and I don't think on (c) it gets to 0.2% difference. I would suggest double-checking this part as it confuses readers at this point. 
* In addition, I think Figure 6 description states (d-e) while it is supposed to be (d-f)
* Another suggestion would be to highlight in each of the tables the best-performing metrics as you use quite a few of them and it becomes more difficult to check. This is a minor suggestion but can potentially improve readability. 




limitations: The limitations were not explicitly discussed with the only suggestion for future work to include automatic switching between the training modes. I think the authors might want to consider unpacking the limitations of this work a bit more.
review_title: null

Meta-review: metareview: The paper identifies and illustrates a practically relevant challenge for training of deep learning-based recommender systems on distributed architectures: switching between synchronous and asynchronous training modes. The proposed mechanism called global batch gradient aggregation (GBA) is simple but mitigates the need to do hyper-parameter tuning when switching which was identified as the critical performance bottleneck. Experiments were conducted on industry-scale recommendation tasks and show that the proposed method is effective and improves in accuracy over fully asynchronous training methods and in speed over synchronous training schemes.

The prevailing opinion among the reviewers was that the paper is well written, well executed and addressed a practically relevant not previously studied problem. The identified challenge is clearly outlined and the proposed solution is well motivated, analyzed with ablation studies and shown to be effective. I advocate acceptance because it is a well executed practical piece of work.

For a potential camera ready version the authors should anticipate the questions of the reviewers and clarify them in the manuscript. They should also be more explicit about the manual steps and the parameter choices required when implementing the method so the limitations are more clear. Further, it would be appreciated if the authors would do an effort beyond acceptance of the paper to push the (currently proprietary) optimizer into an open source framework to make it available to the broader community as promised in the checklist.

confidence: Certain
Decision: Accept