Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers a class of problems called multistage stochastic optimization (MSSO) (also known as stochastic programming) problems, which are essentially MDPs but with linear rewards and constraints. Such problems are very commonly studied in the OR community and solved using a technique called stochastic dual dynamic programming. The main contribution of the paper is a way to learn a neural mapping from MSSO problem instances to value functions, which can be used to warm-start the SDDP solver. Since the output of the neural network is a convex value function parameterized by a list of order invariant affine cuts, the authors propose using an Earth mover’s distance in their loss function.
main_review: Strengths:

-The application to MSSO is interesting. There has been significant attention in using neural methods for deterministic optimization problems (integer programming, for instance). Given the utility of MSSO in OR problems, this learning-based framework could be impactful.

-The use of Earth mover’s distance in learning convex value functions is also interesting.

-There are some strong empirical results, showing significantly improved optimization times with little reduction in performance.

Weaknesses:

-Novelty of the overall framework is relatively limited. As I pointed out above, there have been many papers combining machine learning and existing optimization techniques and therefore I feel that the conceptual novelty of the current paper is low.

-How do RL algorithms, which focus on infinite horizon problems, get ported to the finite horizon case?

-The discussion of MSSO vs MDP is bizarre to me. It is accepted in the OR community that the MSSO formulation is essentially an MDP (see e.g. [1, 2]), where the state is (x_{t-1}, \epsilon_t) and the action is x_t.  In trying to understand the authors’ comment about using feasible sets as states, I’m wondering if perhaps they are working with a definition of “MDP” that doesn’t allow for state-dependent action sets? However, in many definitions of MDP (e.g., Sec 2.1.2 of Puterman’s text [3]), the set of feasible actions is explicitly allowed to depend on the state.

-In the conclusion of the paper claims that it works well in long-horizon settings, but In the inventory problem, problems of horizon 5 and 10 are considered (what is the horizon of the portfolio problem? I could not find it. Parameters for the portfolio problem should also be included in Table 1). In real-world instances, I would be worried that neural SDDP would need to accurately predict too many value functions to be practical. In such settings, perhaps infinite horizon MSSO models would be more appropriate approximation (see, e.g., [4, 5])? Discussion on this point and more evidence would improve the paper.

[1] ​​http://www.optimization-online.org/DB_FILE/2009/12/2509.pdf
[2] https://arxiv.org/abs/1605.01521
[3] https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316887
[4] http://www.optimization-online.org/DB_FILE/2019/09/7367.pdf
[5] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2942921
summary_of_the_review: My assessment is that the paper has a nice application to MSSO problems (which could be impactful due to their usage in OR models), but conceptually the novelty is low. The empirical work shows that the algorithms are generally impressive, but some details are missing. The paper could also use more clarity in writing -- see some of the comments that I brought up above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the neural variant of stochastic dual dynamic programming for solving multistage stochastic optimization, which overcomes the limitation of SDDP. The paper proposes the algorithm $\nu$-SDDP, which continually self-improves by solving successive problems. The empirical investigation shows that $\nu$-SDDP outperforms SDDP and reinforcement learning in several synthetic and real-world process optimization problems.

main_review: Strength:
- The paper is well-written, motivated, and organized. The comparison of MSSO and MDP makes the paper easier to follow for different communities. The motivation of the structure of the value functions is well explained.
- The proposed neural variant of SDDP addresses the exponential dependency of traditional SDDP over the size of the action space. 
- The paper provides a thorough empirical investigation of their algorithm, which shows the advantage of $\nu$-SDDP.

Weakness:
- MSSO problem assumes that the problem context is completely known, while model-free RL does not require any knowledge on the model (such as reward function and transition kernel) and can directly learn from the data. It would be better if the paper could compare $\nu$-SDDP with RL methods with known MDP. 
- Following the previous question, is it possible to solve MSSO via a completely data-driven approach? Specifically, the feasible action set is revealed at time t without knowing $A_t, B_t, b_t$.



summary_of_the_review: The paper is well motivated. The proposed algorithm is novel and interesting. Thus, I recommend acceptance for the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the Multi-Stage Stochastic Optimization Problem. In this "MDP-like" setup, the underlying stochastic process generating state transitions is fixed (independent of actions) but the executed actions determine the subspace of action available in future steps. To solve this problem, the traditional approach is to solve a sequence of LPs, and "backpropagate" dual variables. This process includes heuristical choices of cutting planes. This paper improves uses a data-driven algorithm to improve this process. As a result, instances with a larger number of variables become manageable.


main_review: ** Strengths:

- The paper presents a method that seems sound and non-trivial theoretically with good practical performance.


- Comparisons with RL algorithms are included.


- The paper is relatively easy to follow given its high degree of technical content

** Weaknesses:

- The subject area is of limited significance to ICLR community. MSSO does not have a history at top ML conferences and this paper does little to show it is a mistake.


- The experimental setting is very synthetic. The first problem is completely synthetic and as for portfolio optimization, I doubt there are real-world instances of portfolio optimization that do not optimize a quadratic objective that includes a risk term. This suggests that MSSO as a framework is potentially not rich and flexible enough to spark wider interest.


- Additionally, there are no strong baselines (e.g. coming from earlier benchmark-oriented research) available which makes it hard to judge how substantial the progress is.

** Suggestions

- What I believe would help this paper the most would be to invest in convincing the ML community of the relevance of the MSSO framework. If we can now indeed solve larger instances, can this be applied to any already established RL problems? Or is the linear objective too restrictive? Any steps in this direction would increase my assessment of the paper.
summary_of_the_review: I believe the paper presents a sound and functional method that improves algorithmic performance on the MSSO problem. It is well-written and evaluated.

I remain unconvinced about the wider significance of MSSO but I am willing to listen to new arguments.

Overall, I have a mild tendency to primarily appreciate the technical quality and lean towards acceptance. However, at this point, I do not see myself championing the paper in case of a large variance of opinions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper develops a stochastic dual dynamic programming (SDDP) approach that incorporates a value-function prediction component. Specifically, the authors use opportunistic forward and backward passes to train a neural network model that estimates the value function, which is subsequently used to initialize and guide the basis SDDP iteration procedure. The procedure also leverages the actual value functions SDDP updates to re-train the predictive model for further epochs. Numerical results on inventory and portfolio optimization suggest better error ratios in comparison to SDDP variants and a reinforcement learning model.

main_review: Overall, the paper is clear and investigates an intuitive extension of an SDDP framework with a learning mechanism. In particular, it is natural to consider more sophisticated value-function estimate procedures when the method is inherently iterative, such as in the case of SDDP and other cut-generation procedure methods. The numerical results, while still somewhat restricted to only two benchmarks, provide interesting insights on the trade-off between computational times and solution quality, as well as promising results.

However, I have two major concerns. First, this broader idea of learning from cuts is not new, even within the context of SDDPs. More specifically, I wonder what would be the connections and differences with respect to "batch learning" in SDDPs, which is similar in that it learns a "Q" function while traversing the state paths. Nonetheless, I believe that any other learning model could also be suitable. A recent reference is from EJOR:

http://www.optimization-online.org/DB_FILE/2021/05/8397.pdf

Second, the paper lacks a deeper, more thorough investigation of when the procedure is beneficial or not. It tends to oversell with somewhat bold statements (e.g., "v-SSDP reduces the curse of dimensionality effect," "gain a significant advantage over random initialization" etc), but it is not clear to me whether that should always be the case rigorously. Further, only two benchmarks are not sufficient to make such general statements. 

For instance, what are the properties of the value function (in view of the piecewise elements) so that learning is effective? Could there be a case where the convergence of learning is so slow so that the procedure just adds an overhead? Why only random starts, and not something possibly more systematic but easier to implement (e.g., perhaps https://www.sciencedirect.com/science/article/pii/S0377042715002794?) It would be relevant to precisely identify, at least intuitive, problem structure where we can more easily grasp that random, or any more systematic procedure, could still be beneficial. 



summary_of_the_review: The paper develops an intuitive and natural idea, with promising results on two benchmarks. However, the paper (i) lacks connection with closely related literature (batch learning in SDDPs), and, in my view, (ii) it does not deeply investigate trade-offs of the methodology and underlying structural properties, besides a limited study on the cut generation procedure.
review_title: null

Meta-review: This paper applies deep learning to a problem from OR, namely multistage stochastic optimization (MSSO). The main contribution is a method for learning a neural mapping from MSSO problem instances to value functions, which can be used to warm-start the SDDP solver, a state-of-the-art method for solving MSSO. The method is tested on two typical OR problems, inventory control and portfolio management. The reviewers think that the idea is interesting, the empirical results are impressive, and the paper is well-written. However, there are reservations on its relevance to the ICLR community.
Decision: Accept (Poster)