Kati8/ICLR_2023_2025 huggingface dataset:
authors: Renhao Zhang
authors: Haotian Fu
authors: Yilin Miao
authors: George Konidaris
reviews: 
Comments: This paper applies a model-based RL method to the parameterized action MDP (PAMDP).
Strengths: - The paper is easy to understand.
Weaknesses: - Novelty and contribution is limited.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The paper introduces DLPA - a model based RL approach that brings together parametrized action MDPs with model predictive control. The method was applied to 8 environments that previous works have used and demonstrate up to 30x sampling efficiency.
Strengths: The key difference between previous work and DLPA is that model learning is performed by relying just on the initial state and the actions trajectory instead of all intermediate transitions. This provides better inductive bias for longer time horizon tasks and is show to be a critical component by the ablation study in section 5.3. 

The reported results show that DLPA learns much faster (in terms of number of samples) compared to previous benchmarks which again can be the result of learning longer horizon models.
Weaknesses: The paper brings together PAMDPs and model predictive control in a very conventional way and so I do not find DLPA novel enough from a technical perspective. In terms of results, despite the reported sampling efficiency, even though the testing environments are used by other previous works, they still seem to be relatively simplistic even compared to other game based benchmarks such as ATARI.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: This paper considers reinforcement learning problems in settings where actions are of a mixed type and interactions are readily described by Parameterized Action MDPs (PAMDPs). In this setting, the paper asks whether there exist model-based methods that can efficiently find solutions.

The paper proposes a model-predictive control algorithm called DLPA, and it thoroughly evaluates it against current algorithms on PAMDP baselines.

The paper makes the following contributions.
 * A model-based RL method for PAMDPs called Dynamics Learning and predictive control with Parameterized Actions (DLPA).
 * DLPA is the first model-based algorithm for the PAMDP setting.
 * Empirical evidence that DLPA performs effectively in PAMDP benchmarks.

I really enjoyed reading this paper. Aside from some minor issues, I think the study was well executed and makes clear contribution. I elaborate on my position below.
Strengths: * The paper is generally well written. I was able to easily identify the research questions and understand the main contributions.
* The empirical study is excellent. The experiments are constructed around the standard methodology which is aimed at supporting its main claim for SOTA performance in PAMDP benchmarks. The benchmarks and baselines seem reasonable and fairly chosen. The results provide positive evidence for its SOTA claim and demonstrate a significant performance margin between the proposed method and other baselines. The experiment in Section 5.3 addresses a nuanced claim made in Section 4.1---supporting that with clear positive evidence.
* The paper makes a significant contribution.
Weaknesses: * Many of my comments on this paper are minor.
* The proposed algorithm seems limited to small-horizon problems, as its backward-pass computation and the number of parameters both seem to scale linearly in the horizon length.
* The proposed algorithm has a lot of hyperparameters, which could make the algorithm difficult to tune.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper addresses the problem of model-based reinforcement learning for parameterized action markov decision processes (PAMDPs), by proposing a framework referred to as Dynamics Learning and Predictive Control with Parameterized Actions (DLPA). The core idea of DLPA lies in the integration of neural-network based model learning (transition model, termination model, and reward function) and the model predictive control using model predictive path integral. Extensive experiments against multiple baseline algorithms for PAMDP are provided to demonstrate the efficiency of DLPA.
Strengths: + The proposed framework is technically sound and the key idea of DLPA is clear.

+ Extensive experiments are given against multiple baseline algorithms.
Weaknesses: - The novelty seems rather limited as the main framework of DLPA follows the standard data-driven MPPI, with the contextualization from MDP to PAMDPs. For example, an information theoretic MPC framework was proposed in [G. Williams et al, ICRA 20217] to incorporate the model learning in the MPPI-based planning procedure. 

- Some key information is not provided, e.g. which neural network algorithms are used to optimize Eq. 1?

- No theoretical analysis is given to justify the performance applied to PAMDPs, which seems necessary to help readers understand the significance of the DLPA beyond simple contextualization of data-driven MPPI to PAMDPs. As PAMDPs concerned about finding the right action and the associated parameters to use, is there any guarantee of convergence to the selected parameterized actions during training from DLPA?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This work addresses reinforcement learning in environments with heterogeneous action spaces that can be naturally modeled as Parameterized Action MDPs (PAMDPs). The central research question investigates the existence of sample-efficient model-based approaches for this problem class. The authors introduce DLPA (Dynamics Learning and Predictive control with Parameterized Actions), a model-predictive control framework, and provide comprehensive empirical validation against existing baselines on PAMDP tasks. The paper contributes a novel model-based RL algorithm tailored to PAMDPs, establishes this as the first such model-based approach for this setting, and offers experimental results demonstrating competitive performance on standard PAMDP benchmarks. Overall, this is a well-executed investigation that makes meaningful contributions to an important problem domain.
Strengths: * The manuscript demonstrates strong clarity and organization. The research objectives and key contributions are presented in a manner that is straightforward to comprehend. * The experimental evaluation is comprehensive and rigorous. The study design adheres to established best practices for RL evaluation and directly supports the primary hypothesis concerning SOTA performance on PAMDP tasks. The selection of benchmarks and comparators appears balanced and appropriate. The empirical results convincingly support the SOTA claims with notable performance improvements relative to competing methods. Notably, the ablation study in Section 5.3 provides concrete evidence for the theoretical insights developed in Section 4.1. * The work represents a substantial advancement in the field.
Weaknesses: * Most observations regarding this submission are relatively minor in nature. * The algorithm exhibits scalability constraints for longer-horizon problems, as both the computational cost of the backward recursion and the parameter count grow linearly with horizon length. * The approach introduces a substantial number of hyperparameters, which may present practical challenges in terms of algorithm configuration and implementation.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This work addresses reinforcement learning in settings with heterogeneous action spaces, which are naturally represented as Parameterized Action MDPs (PAMDPs). The central research question investigates whether efficient model-based approaches can be developed for this problem class. The authors introduce DLPA, a model-predictive control technique, and provide comprehensive empirical validation against existing methods on PAMDP benchmarks. The paper's key contributions include: (1) introduction of a novel model-based RL algorithm tailored to PAMDPs called Dynamics Learning and Predictive Control with Parameterized Actions, (2) demonstration that DLPA represents the first model-based approach in the PAMDP domain, and (3) experimental evidence demonstrating DLPA's effectiveness on standard PAMDP evaluation tasks. The work is compelling and exhibits solid experimental design with only minor limitations that do not substantially detract from its value.
Strengths: The manuscript is clearly presented, making it straightforward to discern the research objectives and comprehend the primary contributions. The empirical evaluation is particularly strong, with experiments methodically designed following established protocols to substantiate the claimed state-of-the-art performance on PAMDP benchmarks. The selection of benchmarks and comparative baselines appears balanced and appropriate. Results convincingly support the state-of-the-art claims, showing substantial performance improvements relative to competing approaches. Notably, Section 5.3 provides robust empirical corroboration for the theoretical assertion in Section 4.1. Overall, the work represents a meaningful advance in the field.
Weaknesses: The algorithmic approach exhibits scalability constraints for problems with extended horizons, as both the backward-pass computational requirements and parameter dimensionality increase linearly with horizon length. Additionally, the method incorporates numerous hyperparameters, potentially introducing complexity into the tuning process and reducing practical applicability.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent