Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of placing embedding tables of a recommender system model in a distributed training environment. How the embedding tables are placed can have significant impact on latency. Embedding lookup consists of 4 stages - In the forward pass, the embeddings are calculated and communicated of dense vector to target devices, while in the backward pass, the gradients are communicated to the device with the embedding table and the gradients are then calculated for the embedding table. The tables can easily lead to imbalances if not placed correctly and thus longer latencies during training. Thus, given a set of embedding tables and devices, identifying the best partitioning strategy is important to minimize costs. However, device placement is a NP-hard combinatorial problem. Further,  operation fusion and generalizability requirement further complicate the problem.

This paper proposes DreamShard, a Reinforcement Learning based approach for embedding table placement. DreamShard solves the problem of operation fusion and generalizability using two key ideas - (1) Learning a cost network to directly predict the cost of fused networks (2) Training a policy network that interacts with an estimated Markov decision process (MDP) without real GPU execution. 
strengths_and_weaknesses: Strengths 

1. The results of the paper are competitive with recent state of the art methods and evaluated on strong baselines
2. The idea to capture cost of communication and operator fusion using a learned network to generalize to unseen tables, different number of devices etc can help provide a practical solution to this problem 


Weakness -
None that I can think of.
questions: Questions -

1. Can a table be divided and be placed on > 1 devices in this method? Are there scenarios where a table does not fit in a single device?
2. Can the authors also discuss results comparing a simple greedy based approach? 
3. Is the shared feature extraction MLP and sum reduction in the cost network and policy network sharing weights?
4. Given the placement of the tables is decided by the RL model and can vary across samples during training, how is operator fusion carried out? Is there a compiler pass after placement to determine what operations on a device can be fused?

limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a RL-based approach for embedding table placement on multiple devices (mostly GPUs), which outperforms human expert and RNN-based baselines.
strengths_and_weaknesses: Strengths
* The problem is important and rarely exploreded before
* The proposed method seems reasonable to me, with lots of reasonable heuristic baselines
* The writing is clear and the paper is easy to follow 
questions: * Could the method scale to larger settings? Currently 4 gpus are being used.
* In a real world setting, the vocab distribution constantly changes (e.g. new popular items) which may affect the load balance. Could the method adapt to this dynamic setting?
 
 

limitations: no.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the embedding table placement problem for distributed recommender systems, where the the operation fusion and the generalizability is important and challenging. For solving the problem, the paper formulates the table placement process as an MDP, and proposes an RL framework named as DreamShard which consists of a cost network and a policy network. Extensive experiments show that DreamShard performs well.

strengths_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow.
2. The embedding table placement problem is practical.

Weaknesses:
1. The introduction of the related work is not enough. I am not sure about the novelty of this paper.
questions: 1.  Does the MDP formulation is usual in the related research? 
limitations: I have  no other specific suggestions about the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents DreamShard for embedding table placement in recommender systems to balance the computation and communication costs. The authors formulate the table placement process as an MDP and train a cost network to estimate its states and rewards. Then they update the policy network by interacting with the estimated MDP. The results show speedup over the existing algorithms and good generalizability.

strengths_and_weaknesses: ### Strengths:
1. The paper is clearly written and very structural. The idea of formulating the table placement process as a MDP is interesting.
2. This paper considers the different speedups across different table combinations and proposes a generalizable embedding table placement problem.
3. This paper addresses the embedding table placement problem with two networks, achieving good performance in speedup and generalization. The experiments are well-designed and thorough.

### Weaknesses:
1. It would be nice to have more results on DLRM with 8GPUs in table 1.
2. Why the author choose 21 table features such as hash size? Can it influence the performance of the model?
3. Lack of comparison with related works, e.g., RecShard.
questions: Pleae see comments above. 
limitations: The authors have discussed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper proposes DreamShard, a RL-based framework for placing embedding tables across multiple devices in distributed recommender systems. DreamShard jointly trains a cost model (to predict the cost of communication and operator fusion for new configurations) and a policy network to make placement decisions based on the cost model. This two step design makes the algorithm more efficient than naive RL solutions and end-to-end training leads to better generalization than model-based offline strategies.

All reviewers agree that the paper is well written and proposes a practical solution to an important problem that is not well studied in the literature. Furthermore, the paper has a strong empirical section that compares DreamShard to strong baselines on open-sourced and production datasets, shows good results and conveys a broad picture of many aspects of their method. 

Overall this is a very well executed paper proposing an efficient and practical solution to an underexplored problem. I recommend acceptance. 

For the camera ready the authors should include the new scaling experiments they performed to address the reviewers comments. I would also recommend integrate some of the clarifications regarding the contributions and distinctions from prior work (comments to Reviewer AKvA) in the paper.  Also, it might be worth including the greedy baseline numbers for some experiments, just to put the performance into perspective.
confidence: Certain
Decision: Accept