Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zhengye Han
authors: Yusen Huo
authors: Zhijian Duan
authors: Tianyu Wang
authors: Yeshu Li
authors: Zhilin Zhang
authors: Chuan Yu
authors: Jian Xu
authors: Xiaotie Deng
reviews: 
Comments: The paper proposes a new method for offline reinforcement learning. Instead of learning Q-values or update policy in a conservative mode, the approach 'imitate' the trajectories, in which actions are normalized and weighted. With this simple approach, the paper shows that the proposed method has reasonablly good performance on D4RL benchmark and especially better performance on budget allocation environment.
Strengths: 1. The proposed method is novel to me, as it abandon many techniques for conservative value or policy learning, and propose to update a loss function that with normalized actions and weights.
2. The empirical results show superior performance on budget allocation environment.
Weaknesses: 1. The presentation is bad. In general, the notation is confusing and it is hard for me understand the method from the beginning. y(a), \hat{a} are not clearly representative of their meanings and there is abuse of notations that are not reasonable.
2. The connection of the method with the classic policy update is not straightforward to me. That is to say, sec 3.2 is not informative. One should interprate the weights in a more intuitive way.
3. The empirical results is not compelling, especially there are new hyper-parameters being brought in the method. It is unclear if the approach is sensitive (or not) to hyper parameters. This is important to empirical offline RL algorithms in practice. Furthermore, the performance is not persuasive on D4RL benchmark.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This work deals with offline RL.  It presents a policy gradient method that performs planning the next sequence of actions in a higher dimensional representation space.  

The first few modifications are relatively basic, they first learn the policy using a cross entry loss, to optimize the action sequences.  They then introduce a cross entropy policy gradient loss.  A KL term is added to keep the target action distribution and policy action distribution close.

A neural network is introduced that outputs a sequence of actions per each time step and predicts the mean and standard of each action step in the sequence.  The target action distribution is then computed and the policy parameters are updated using the cross entropy loss with the targets

The method is evaluated on D4RL and they also show results that support the cross entropy loss having more stable gradients than the LLH loss.
Strengths: * This paper takes a bold approach at offline RL, taking a drastically different approach to policy gradient methods.
* The multi step planning approach has good justification in many other works.
Weaknesses: * This is a highly unusual format and I question the validity of this approach.  I would appreciate if the authors derive the policy gradient theorem under this objective.  I would also like to see more theoretical or empirical support for this since it greatly differs from standard policy gradients.  I see many problems where this loss would fail to be efficient or stable.  While the gradient may show stability in one experiment I would like to see stability of the method for offline RL learning.
* The method fails to outperform currently offline RL algorithms in a large portion of tasks and is only evaluated on one environment.  In some environments, the planning method is dramatically worse than CQL.  There are only modest improvements in a few tasks.
* The method is only evaluated on D4RL.
* The paper is relatively hard to read and the changes to the loss need further justification.  

I recommend rejection of this work given the limited empirical results and the bold changes to standard RL that are not completely justified in the text.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper argues that deciding on a single action step-by-step may be detrimental to long-horizon or horizon critical tasks. Thus, it proposes to plan a sequence of actions for multiple steps at a time. In addition, it suggests to replace the traditional maximum likelihood based loss function with the cross-entropy loss commonly used in supervised learning for better stability. They perform experiments on couple of environments and demonstrate improved performance.
Strengths: 1. This paper presents an interesting idea and considers that for long-horizon tasks the agent should plan a sequence of actions instead of traditional single action. Their motivation came from modeling the temporal dependency which is a crucial aspect in sequential decision making. The hyperparameter k, denoting the length of the predicted action sequence, provides an opportunity to control the granularity of making a decision. Careful tuning of this variable indeed can lead to improved performance. 

2. The cross-entropy (CE) loss, although is not a novel one in RL, the use of that with multi-step action prediction is somewhat new direction. I appreciate the author's effort to align CE loss with general policy gradient objective. The use of $G_t$ in the CE loss facilitates achieving core RL objective by preferring actions with higher returns.

3. Experiments with the custom Budget Allocation Environment add values to the validation. After the methodology secrtion, I was interested to see such experiments with varying trajectory length. The proposed approach performs better with increased trajectory length.
Weaknesses: 1. One of my main concern is related to the possibility of overfitting. The policy learns multiple actions for consecutive time-steps just based on a single observed state (the model sees every k'th state as depicted in figure 2). Further, the model incorporates KL divergence based regularization. Such a term that reduces the discrepancy between the policy and target distribution across the full trajectory may lead to memorization of the training trajectories. However, the paper doesn't investigate the generalization performance in-depth.

2. The comparison between CE loss and MLE loss is not fully comprehensive. In figure 3, the y-axis denoting gradients have different scale. Thus, the comparison is not fair. A more zoomed-in look at the CE gradients [1.0-1.5] may reveal otherwise. 

3. This paper doesn't include rigorous ablation studies to properly attribute performance gain to the introduced components. Experiments with removing KL divergence based regularization, action normalization, etc. one at a time would be helpful.

4. The paper should include more information regarding the hyperparameters, components, and  experimental settings for reproduction and clear understanding. For example, there is no discussion regarding the position embedding as illustrated in figure 2. Also, the related work section needs improvement.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3