Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This tiny paper uncovers the relation between Soft Q-Learning and Soft Actor-Critic framework optimization based on maximum entropy framework. Convex optimization method is utilized to connect the optimal value function and policy between these two RL frameworks. This analysis method can be further leveraged for other regularizer.
strengths_and_weaknesses: - Strengths:
  - Clarity: The proof process is succinctly and clearly described in the paper, short but efficient enough.
  - Correctness: The analysis strictly follow mathematical analysis with correct theorem in SAC and SQL frameworks, thus the conclusion is correct both theoretically and intuitively.
  - Reproducibility: The lemma and theorem in the paper are all described and proofed in detail. It is easy to reproduce the  conclusion of this paper.
  - Follows basic requirements: This tiny paper follows the all ICLR submit requirements.
- Weaknesses:
  - The writer mentions the  same analysis procedure can also be applied with other regularizers without a deeper illustration .  
suggested_changes: Give an illustration to the generalizability of the proposed theorem. 
- Is the process of changing LogSumExp optimization to repeated policy evaluation/improvement generalizable? Can it be applied to transform all the arduous LogSumExp optimization problem into an easier one that is not only in RL setting of SQL/SAC?
- How this introduced analysis procedure can be applied for other specific regularizer like KL divergence. Besides give a specific different form of $\Delta(s)$, how can it help with current RL optimization method?
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This paper provides theoretical proof that soft actor-critic converges to the same solution as soft Q-learning in the limit. This has the potential to aid and ease optimization and can potentially extend to other regularizers to provide the same benefit.
strengths_and_weaknesses: Strengths:

- Clarity: Relevant literature is discussed, especially those that were used to build the proofs shown in this work. 

- Correctness: Proofs appear to be correct. Showing that the Q functions and policies are equal proves equal convergence in the limit.

- Reproducibility: Assumptions and proofs are provided, variables are defined well, and the proofs provided can be followed and reproduced. Preliminaries are well stated.

- Follows basic requirements: Is within the page limits and adheres to the code of conduct

Weaknesses:

- Clarity: While it is demonstrated that these converge to the same solution in the limit, I think it would be great to have some additional details on when the convergence may actually happen and some implications of this result from a real-world training scenario.
suggested_changes: Thank you for your submission! Overall, I think this paper did a great job making the proposed proofs understandable and clear. 

My one suggested change for this paper would be to include more details on the implications of the proofs shown here. I think including some details on how this may impact future training schema and what recommendations you have (e.g., should SAC or SQL be used?) based on what you found here would substantially increase the clarity of the paper.
review_title: null

Decision: Invite to present (notable)