Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers gradient descent algorithms for computing a parameter of the optimal policy according to a possibly unstable trajectory of a Markov decision process. It is shown that under technically strong assumptions such as continuity, convexity, and smoothness, the rates of convergence will be linear, assuming that the horizon is short enough. 
strength_and_weaknesses: I like the general idea of the paper; it has an understandable approach and puts forward an interesting and to some extent important problem. The writing in most of the manuscript is acceptable, but some improvements help. The technicalities seems based on strong assumptions though.

The deterministic nature of the dynamics restricts applicability of the approach to data-driven problems. Also importantly, in case of randomly or adversarially disturbed dynamics, the stabilization needs to be different since other than the initial state, the innovation can cause instability. Although the authors claim that the method is applicable to disturbed dynamics, the analysis does not seem so. Further, the existing literature of learning-based stabilization is not covered (enough). The setting and analysis of the GD does not seem innovative and seems a little artificial. The enumeration looks strange. In some places the language is not articulate enough, e.g., in Lemma 3.7. 

The technical assumptions, e.g., 3.3 and 3.4, are a little restrictive and seem consequential. So, I think further justifications and intuitive arguments are needed. Importance of the main results Thm 3.9 and 3.10 are not clear, and it is also unclear what technical difficulties need to be addressed to obtain them. Finally, considering the step size upper bounds, applicability of the algorithms to only small time horizons renders the framework limited in the sense that the step size will be so small that the method will have no superiority to other learning based stabilization approaches, even if it is somehow applicable. 

The edits the authors provided in the resubmitted version do not seem adequate. I suggest the authors to do further rewrites and to go deeper for proving interesting results.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: I prefer to give the authors a chance to see how they can convince the reviewers that their paper is interesting, important, and innovative. 

--------------------------------
post rebuttal: this reviewer carefully read the rebuttal and updated the review.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the gradient-based optimization for a special branch of RL problems. Due to the unstable nature of the system, small deviation leads to exponentially growing effects on the state
evolution trajectory and the reward/cost function, which raises issues for gradient-based optimizations. The authors proposed two methods to alleviate the effect of instability and their effectiveness is validated from
both theoretical and experimental points of view.
strength_and_weaknesses: Overall, the paper seems to contain interesting results. 
Some comments to improve the paper are summarized below:
1) For an MDP with finite and discrete state and action spaces, the stable MDP is not clear. 
Only for MDPs with continuous state and action spaces, that are main focus of the paper, stability is meaningful. 
In abstract and introduciton, the system is not clearly indicated, and hence, it is very confuzing at the beginning on what stability means for MDPs.
Therefore, it would be better to indicate that the system is an MDP with continuous spaces at the beginning. 
2) In assumption 3.2, it is not clear what is the basis function. In the explanation after assumption 3.2., it says that || A-BK||^2 corresponds to \phi. 
However, || A-BK||^2 is scalar number, while \phi needs to span the function space. This discussion seems to be unreasonable. 
3) The convergene results in Theorem 3.9 and Theorem 3.10 are not clear. It is not clear why the inequality leads to convergence. Some improvement of presentation may be needed.
4) It would be better to indicate earlier in section 3.2.2. that the results are all for deterministic settings.
clarity,_quality,_novelty_and_reproducibility: The paper seems to include novel results. 
The presentation needs to be improved as mentioned in the previous comments.
summary_of_the_review: This paper focuses on the gradient-based optimization for a special branch of RL problems.
Overall, the paper seems to contain interesting results. 
The presentations need to be improved as mentioned in the previous comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes the convergence of policy gradients on I/O unstable problems, in particular unstable LQR problems. Under a variety of assumptions, the authors claim to proof an bound on the learning rate, related to the largest spectral eigenvalue of the dynamics matrix of LQR. Experiments show that a log mapping of the objective stabilizes gradient descend on LQR's policy matrix K.
strength_and_weaknesses: I have to admit that I did not understand this paper. In particular, I cannot say how realistic Assumption 3.2 is, and how Theorems 3.14 and 3.15 proof stability. However, the experimental results look good, even though I find it hard to say how general they are.
clarity,_quality,_novelty_and_reproducibility: The connection with the LQR controller could be better clarified for a less involved audience. The paper motivates the use of LQR controllers, but then seems to prove theorems that are more general. 

I also did not understand how the theoretical insights (for example the log-transformation) are translated into Algorithm 1 and 2. Alg.2 seems to be gradient descend on standard LQR, whereas Alg.1 seems to do something similar for general differentiable transition functions some additional spectral norm regularization. However, it is not clear to me how this is related to the convergent guarantees of the theorems above.
summary_of_the_review: I admit that I was not able to follow the paper. I am uncertain whether this is the fault of the paper or mine. While I recommend to reject the paper, I am happy to change my mind if more qualified reviewers think the paper deserves publication.

**Post-rebuttal**

I thank the authors for their attempt to explain their paper, but I still feel that I do not understand the paper enough to give a certain recommendation. Due to this uncertainty I retain my previous recommendation to reject the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the policy gradient method of unstable dynamic systems. The stability considered in this paper is the so-called "input-output" stability (I/O), where a function of the system state is always bounded by a function of the initial state and time. The paper considers gradient descent algorithms for computing a parameter of the optimal policy according to a possibly unstable trajectory of a Markov decision process. It is shown that under technically strong assumptions such as continuity, convexity, and smoothness, the convergence rates will be linear, assuming that the horizon is short enough.

Strengths:
- it has an understandable approach and puts forward an interesting and, to some extent, important problem

Weakness:
- contains a series of strong technical conditions, which might not be realistic
- presentation is not clear enough: it is hard to get a clear understanding of why the instability conditions could lead to difficulties in the algorithm and analysis. There is a lack of discussion on the input/output stability (it looks like ISS is a special case of I/O stability, and these functions $\beta$, $\gamma$ are also not clear, can they be arbitrary functions?). The paper also contains a number of typos that make it hard to understand what is going on.

The metareviewer suggests the authors to address the concerns from the reviewers and improve the paper accordingly.

justification_for_why_not_higher_score: There are a number of weaknesses presented in the paper, and the authors failed to convince the reviewers that the paper contains significant contributions to be published in ICLR.
Decision: Reject