Venue: UAI 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., lack of novelty, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: Q1 Summary and contributions: This work focuses the resource allocation problem where its goal is to ensure feasibility with respect to constraints and risk-based objectives. Specifically, the neural-progressive hedging (NP) framework is proposed where it trains an unconstrained RL first, then a stochastic program is used to re-optimize the policy under constraints and risk measures during the online execution. Finally, experiments on two real-world tasks have been conducted, with results demonstrating its effectiveness.

Q3 Main strengths: - The resource allocation problem is closely related to the real-world applications, which would be interesting to both the theoretical and applied researchers in RL community. It is novel to propose the neural-progressive hedging framework to solve this resource allocation problem under constraints and risk measures.
- The methodology is generally sound to me. Besides, this paper also makes the theoretical analysis for the proposed method.

Q4 Main weakness: - In the section of introduction, this paper claimed that there are three shortcomings, which impede the uptake of RL for real-world problems. However, this claim only holds for the constrained or risk-sensitive RL problems rather than the "general" RL problems.
- The comparison for sample efficiency is unfair. It looks NP has an ensembled model for handling different scenarios while DDPG is often single model. Therefore, the comparison should be counted in terms of the total number of samples required for each model rather than the time steps.
- Current results didn't support the claim on better explainability of the policies. It looks the number of scenarios is more like a hyper-parameter.

Q5 Detailed comments to the authors: - See my comments in the main weakness.
- -Both the method and the theoretical analysis are based on the assumption that $\tilde(f)$ is a convex function. However, there lacks the discussion on how to garantee the convexity under the real-world setting. Beside, NP relies on the learned model and assumes this model should be good approximations to the true state transitions. How to garantee a sufficiently good model under the real-world setting?
- What does the "scalable manner" in this paper refer to?
- What does the "unseen scenarios" in this paper refer to? Since NP relies on a good model, the transitions should be known and keep fixed. It looks to me that this unseen scenarios refer to the partially observable states, is it true?
- From Figure 4, it looks the more scenarios, the better performance is. Is this true?
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: Considering the current form of the paper, it feels to me that the main weaknesses outweigh the main strength a little bit. Therefore, I prefer to borderline reject.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high to excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: The paper proposed a framework, called neural-progressive hedging (NP), that leverages stochastic programming during the online phase of executing a reinforcement learning (RL) policy.
Q3 Main strengths: The paper is relevant to AI.

The conceptual approach seems to be technically sound, with no obvious flaws in it.

The claims that the proposed algorithm is an improvement over existing ones is supported by theory and experimental results.

The text is clear and well written. The paper is well organized, with well-structured sentences, where we can understand the development of the argument well. The abstract gives a good description of the paper, summarizing it well. Figures, diagrams and formulas are all readable and formatted correctly.

Q4 Main weakness: The research methodology could be improved – I do not think it is possible to reproduce the experiments using only the information in the paper (an important point in scientific research). But I am not sure there would be space for this information.
Q5 Detailed comments to the authors: The approach to solve the proposed problem is novel, seems sound, and will have an impact in the field. It is well written, clear, and looks reproductible. 

Therefore, the paper should be accepted.

Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: A very theoretical paper, with very good results
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high to excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: This paper proposes a modification to existing RL methods where the goal is to
ensure that constraints are satisfied and also feasibility of risk measures.

Q3 Main strengths: I'm afraid that, as told to the conference organizers, this paper is far from my
field of expertise and thus, all I could do is to verify that there are no typos
and that it is seemingly well written.

As a foreigner, though, the paper looks interesting and I assume it is novel,
but I could not assess on the novelty or importance of the methods employed
here.

Q4 Main weakness: I'm afraid that, as told to the conference organizers, this paper is far from my
field of expertise and thus, all I could do is to verify that there are no typos
and that it is seemingly well written.

As a foreigner, though, the paper looks interesting and I assume it is novel,
but I could not assess on the novelty or importance of the methods employed
here.

Q5 Detailed comments to the authors: I'm afraid that, as told to the conference organizers, this paper is far from my
field of expertise and thus, all I could do is to verify that there are no typos
and that it is seemingly well written.

As a foreigner, though, the paper looks interesting and I assume it is novel,
but I could not assess on the novelty or importance of the methods employed
here.

Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: Just guessing, as I repeatedly said this paper falls very far from my field of expertise
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: This paper studies the problem of learning an optimal policy in a Markov Decision Process (MDP) model which satisfies constraints such as conditional value-at risk (CVaR) during the execution of the policy. The proposed method leverages stochastic programming during the online phase of executing a reinforcement learning (RL) policy.
Q3 Main strengths: This paper studies an interesting problem. The problem of learning an optimal policy subject to some budget constraints seems relevant to many practical challenges, e.g., the class of resource allocation subject to structural constraints during policy execution. 

The proposed method in Algorithm 1 appears to be technically sound. The key to Algorithm 1 replies on a novel offline RL optimization procedure. Theorem 1 ensures that it converges to the optimal solution under some reasonable convex assumptions. The authors performed extensive experiments. The simulation results supports the efficiency of the proposed approach.
Q4 Main weakness: The proposed algorithm is presented as an online RL algorithm. However, its exploration strategy seems a bit naive. It appears to be a greedy algorithm that always performs an action that appears the best from the empirical estimates. It has been known that the greedy strategy does not perform well in the online setting. I would like to see the online analysis of Algorithm 1, e.g., showing that its cumulative regret converges as the number of episodes T grows infinite.
Q5 Detailed comments to the authors: Please see Q4.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: This paper studies an interesting problem of sequential resource allocation with respect to typical resource constraints. It captures challenges in many practical applications including portfolio optimization with liquidity constraints for financial planning. The proposed method seems reasonable and could converge to the optimal solution. While the online regret analysis is missing, the offline RL component appears novel and could be useful for the AI community.
review_title: null

Meta-review: Meta Review: The authors have provided a strong rebuttal, which they are encouraged to incorporate into their paper on revision to clarify concerns raised by reviewers.  While one reviewer retains their concerns after the rebuttal, three other reviewers side with acceptance.
Decision: Accept (Poster)