reviews: 
Comments: This paper proposes Action Sequence Planner (ASP), a novel framework for offline reinforcement learning. ASP replaces the commonly used likelihood maximization loss in policy gradient methods with cross-entropy loss, resulting in stable training and consistent gradients. Rather than predicting actions step-by-step at each state, ASP directly predicts an entire action sequence using fully connected neural networks, avoiding the need for more complex architectures.
Strengths: - The main idea of the paper—using cross-entropy loss to replace the commonly used likelihood loss—is both interesting and novel, with significant potential to open a new direction for offline RL.
- The novelty of the proposed approach is evident.
- The experimental results demonstrate the advantages of the proposed method effectively.
Weaknesses: ## Technical Issues
- The proposed method claims to address or relax the issue of inaccurate value estimation caused by using value functions to bootstrap future returns. However, throughout the paper, the only approach taken to achieve this seems to be the replacement of the value/advantage function with the discounted return, which appears insufficient. As discussed in work like GAE [1], multiple design choices could serve as learning objectives or weigh the action reproducing likelihood. The use of a value function, rather than the true return, helps avoid high variance caused by different rollouts. Directly using the true return may increase variance, potentially harming training stability, which weakens the technical justification.

- From a deep learning perspective, using fully connected (FC) layers to directly generate trajectories or sequences is not technically sound, due to their poor scalability to sequence length, with complexity $O(N^2)$. Generally, sequence models like RNNs and Transformers are preferred for sequence modeling. The proposed method contradicts this intuition without providing strong arguments or experiments to justify why FC layers are chosen over alternative models that might better fit sequence modeling tasks.

- The mapping from the action trajectory to its distribution in line 189 may lead to constant shift risks. For instance, if a constant is added to all actions in a trajectory, the distribution remains unchanged. Using such a distribution in the learning objective could introduce a constant shift in the learned action trajectory. Please analyze this potential risk.

- The trajectory distribution calculation in line 189 does not account for the dimensionality of actions. For tasks with high-dimensional action spaces, how should this distribution be computed?

- The content between lines 227 and 230 is unclear. Please consider restructuring it for clarity.

- Line 231, if $y_t(a)$ is a scaling factor that adjusts each action’s contribution to the gradient update, does this mean that the action with the minimum value will always have zero contribution, as its corresponding $y(t)$ would be zero?

- In line 236, why does incorporating actions that are deemed more significant by the normalized distribution benefit the policy? The causality here is unclear, as the distribution is merely a numerical representation of the action value, not a meaningful distribution that captures the reproducing capability. Please provide further explanation.

- In line 251, why is an off-policy method mentioned here?

- The proposed method, particularly the trajectory distribution matching component, is technically quite similar to Behavior Cloning (although BC is imitation learning rather than offline RL). What causes the ablated method, ASP-MLE, to consistently perform worse than BC in Table 1? Is the setting for ASP-MLE fair, given that it uses hyperparameters from a model trained with a different loss function? Notably, the gradient magnitude differs by $10^4$?

## Writting
The storyline of the paper feels somewhat loose and could benefit from proofreading. Many arguments and conclusions are presented without sufficient supporting evidence or references to relevant literature.

## Missing discussions to literature works
Several RL papers [2, 3] relate to action sequence generation and might offer alternative design choices beyond using FC layers. Please consider discussing and comparing these methods.

## Minor
Please number key equations in the paper for readability.


## References
[1] Schulman, John, et al. "High-dimensional continuous control using generalized advantage estimation." ICLR 2016.

[2] Raffin, Antonin, Jens Kober, and Freek Stulp. "Smooth exploration for robotic reinforcement learning." Conference on robot learning. PMLR, 2022.

[3] Zhang, Haichao, Wei Xu, and Haonan Yu. "Generative planning for temporally coordinated exploration in reinforcement learning." ICLR 2022.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 1
Clarity: 2