Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper Summary
This paper investigated the problem of scheduling and resource allocation for a time-varying set of clients with heterogeneous traffic and QoS requirements in wireless networks. It proposed to solve this problem with distributional based DDPG with Deep Sets, and conducted experiments showing performance gains against conventional methods.

Paper Strength
1.	The paper considered a complex scheduling scenario, which is a hard problem by conventional optimization methods. The problem setting takes into account traffic model, geometry model, channel model, and rate model. Both full-CSI and partial-CSI scenarios are considered. 
2.	The paper adopted state-of-the-art techniques and works fine. Specifically, Distributional RL and Deep Sets for speeding up the convergence and reducing neural network parameters, respectively. 
3.	The proposed algorithm outperforms conventional combinatorial optimization methods.

Paper Weakness
1.	The presentation of the paper should be improved. Right now all the model details are placed in the appendix. This can cause confusion for readers reading the main text. 
2.	The necessity of using techniques includes Distributional RL and Deep Sets should be explained more thoroughly. From this paper, the illustration of Distributional RL lacks clarity.
3.	The details of state representation are not explained clear. For an end-to-end method like DRL, it is crucial for state representation for training a good agent, as for network architecture.
4.	The experiments are not comprehensive for validating that this algorithm works well in a wide range of scenarios. The efficiency, especially the time efficiency of the proposed algorithm, is not shown. Moreover, other DRL benchmarks, e.g., TD3 and DQN, should also be compared with. 
5.	There are typos and grammar errors.

Detailed Comments
1.	Section 3.1, first paragraph, quotation mark error for "importance".
2.	Appendix A.2 does not illustrate the state space representation of the environment clearly.
3.	The authors should state clearly as to why the complete state history is enough to reduce POMDP for the no-CSI case.
4.	Section 3.2.1: The first expression for $J(\theta)$ is incorrect, which should be $Q(s_{t_0},\pi_\theta(s_{t_0}))$.
5.	The paper did not explain Figure 2 clearly. In particular, what does the curve with the label "Expected" in Fig. 2(a) stand for? Not to mention there are multiple misleading curves in Fig. 2(b)&(c). The benefit of introducing distributional RL is not clearly explained. 
6.	In Table 1, only 4 classes of users are considered in the experiment sections, which might not be in accordance with practical situations, where there can be more classes of users in the real system and more user numbers.
7.	In the experiment sections, the paper only showed the Satisfaction Probability of the proposed method is larger than conventional methods. The algorithm complexity, especially the time complexity of the proposed method in an ultra multi-user scenario, is not shown. 
8.	There is a large literature on wireless scheduling with latency guarantees from the networking community, e.g., Sigcomm, INFOCOM, Sigmetrics. Representative results there should also be discussed and compared with. 

======
post rebuttal: My concern regarding the experiments remains. I will keep my score unchanged. 

review_title: An interesting application of DRL, but the paper could be improved. 

final_score: rating: 7: Good paper, accept
review_content: This paper addresses the long standing problem of scheduling and resource allocation in wireless networks using modern Deep Reinforcement Learning techniques. 
It is clearly written and easy to follow but suffers from several minor typos.
The methodology is well justified and thoroughly motivated.
Experimental evaluation seems thorough and provides convincing results.

The MDP is not described thoroughly enough: 
What is your reward, action space, state space, observations?
How is the allocation deadline incorporated into the reward?
It would be nice to have these details listed in a sub-section somewhere in Section 3.

Regarding the evaluation, "synthetic" traffic patterns are used.
Can you use real world traces with a simulator for evaluation (similar to https://github.com/hongzimao/pensieve)?
Also real world applicability is not addressed?
Will the inference times for the deep network lead to any significant overheads when measured at the time scale of wireless communications?
Overall, the evaluation setup seems preliminary to me and needs more work to provide assurance of real world usability.
review_title: Promising Application of DRL to a classic wireless problem

final_score: rating: 7: Good paper, accept
review_content: Basically, it seems that the proposed method is interesting and meaningful. The scheduling problem in this paper is based on the analogy to a server having a water pitcher, and the deep reinforcement learning approach for the scheduling problem has been designed. However, the scheduling problem in wireless networks is a very famous issue. Of course, applying DRF to it is quite interesting. However, the authors need to describe the conventional well-known scheduling algorithms and compare them with the proposed scheme (now, the current paper only focuses on applying the DRF to the scheduling and evaluating its performance in aspects of an optimization problem.). Further,  typically, in scheduling problems, efficiency (total data rate) and fairness are the key factors and it is needed to describe the relationship between these conventional performance metrics and the satisfaction probability. 
review_title: This paper investigates the deep reinforcement learning method to schedule the traffics with heterogeneous requirements under dynamic channel environments. The proposed method is compared with its lower and upper bound methods, i.e., so called myopic Knapsack and oracle ILP. It is clear that the proposed method has the merits compared with the lower and upper bound methods in terms of performance and implementation.

final_score: rating: 3: Clear rejection
review_content: The authors propose a deep RL solution for the communication problem of user scheduling and resource allocation. The deep RL solution uses deterministic policy gradient + quantile regression + dueling + deep sets and the authors demonstrate it outperforms classical solutions on benchmark tasks. 

Recommendation:

I quite frankly have no prior knowledge of the task this paper aims to solve. It didn't come across as a grandstanding AI challenge in any capacity (feel free to debate this) so I'm looking at this paper from the angle of significance to the deep RL community. In general, the questions I'm asking are: Does this paper introduce a novel deep RL algorithm? Is the knowledge produced by this paper generalizable to other problems or algorithms? Does this paper provide value to anyone who isn't concerned with the specific application? 

As of now, I feel like the answer is no to all three and so I would recommend rejection to this conference. However, there are presumably venues which are a better fit and I hope the authors consider submitting elsewhere. 

There were other issues with the paper. In general, clarity and organization were a big issue for me. The authors were very rigorous with their supplementary material, so I believe most of the information is there, but was not presented in an easily digestible manner. 

Strengths:
- Thorough supplementary material and code is provided. 
- The use of deep RL to tackle the problem felt well-motived and a good fit. 
- The performance of the agent seems strong but I'm not clear on the significance of some of the results. 

Weaknesses:
- I think the problem set up was clear in the sense that I understood the overarching objective. However, the specifics of the problem, specifically in the context of RL was not. The paragraph about 3.2 is a generic description of the RL problem and left me wondering the connection to the actual application. I realize many of the details are contained in the supplementary material but the statement "the problem can be modeled as an MDP" was not defended & the following description did not clarify the problem statement. For example, immediately after, in 3.2, "high variance randomness" is discussed but its not clear to me why this is the case or how this randomness affects the problem- reward? transitions?
- The structure of the paper did not feel helpful to me. 3.2 is categorized into "Policy Network" and "Value Network" for somebody who is comfortable with RL a lot of the details felt unnecessary but more importantly, this organization doesn't provide a solid presentation of the algorithm. Somebody who is interested in the application and is not an RL expert, won't necessarily follow why "Policy network" is being presented or what that even means necessarily. There isn't a clear overview of the algorithm. 
- Novelty of the algorithm is low in the sense that it is a combination of prior, existing ideas. 
- I felt like many of the algorithmic choices were not well-justified. For example, the use of QR is justified by an analogy? The use of the dueling architecture also seems unusual when the authors also propose a much simpler solution. It was also unclear what the issue with "The main problem was that the distribution Z was far away from 0 making it very difficult for the policy network to well approximate them" exactly meant. 

Minor Comments:
- There are a few latex issues with reversed quotations. 
- "In Figure 2 we provide additional element to support the choice" -> an additional
- The objective of Figure 2 is nice but its confusing to have two sets of experiments presented in the same graph. Class is not explained in the description of the graph and the significance of the graph is not explained in the figure description.

**Post-Rebuttal

I appreciate the authors taking the time to respond. Unfortunately, my belief that this paper is not strong enough from the deep RL perspective to warrant acceptance has not changed. If other combinations of tools do not work, then the authors should improve this justification, with ablation studies or stronger theoretical motivation. I’ll add that my score is not influenced by my concern that this paper may not be a good fit for ICLR (I leave that choice to the AC).
review_title: Interesting application but has clarity issues and perhaps not a good fit for ICLR

Meta-review: The reviewers mostly agree that this paper presents a new deep reinforcement learning-based approach to solving a challenging problem in the communications domain -- wireless scheduling. However, the main concern, expressed almost unanimously, is about the novelty of the ideas in the paper beyond the assembly of existing deep RL techniques and the translation of the scheduling problem to the language of MDPs in a careful manner that respects modern communication systems standards such as 5G (e.g., URLLC and eMBB traffic demands). A secondary concern, also expressed during the author rebuttal discussion, is about adequate comparison to competing approaches motivated from the literature in wireless scheduling. In view of these issues, I suggest that the author(s) explore more appropriate avenues to submit this piece of valuable translational work, including venues that address the specific topic of wireless communication where a more comprehensive evaluation and comparison could be possible. 

(NOTE: The comments and evaluation above disregard the "enhanced" draft submitted by the author(s) during the rebuttal phase. I was informed that the submission was reverted to the original draft due to space constraints being exceeded in the enhanced version.)

Decision: Reject