Title: Scaling Offline RL via Efficient and Expressive Shortcut Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

shortcut models
Abstract: Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline RL remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models – a novel class of generative models – to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL supports both sequential and parallel inference scaling by using the learned Q-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of applying diffusion and flow models to the offline RL problem. The authors introduce a method called Scalable Offline Reinforcement Learning (SORL) which leverages *shortcut models* (a novel class of generative models) for scalability in training time and inference time. The authors establish theoretical results that the shortcut models output a distribution that is close (in Wasserstein) to the data distribution (and thus, that the offline RL policy is regularized towards the behavior policy). Experiments are conducted.\n\nSTRENGTHS AND WEAKNESSES:\nAt a high level, this paper amounts to \"importing\" shortcut models into RL by using them as a policy class in an otherwise standard RL pipeline. Despite the lack of algorithmic novelty, the approach is well-motivated and the authors are convincing about the benefits of this approach, namely that it can simultaneously give expressive multi-modal policies, provide efficient training, and achieve inference-time training. The experimental results are mixed but mostly strong and the theoretical analysis of shortcut models is novel (from what I can tell) and may be of independent interest. The shortcut model analysis itself is not a complete analysis of the SORL method (it only analyses the regularization term by itself). For the experimental results, it would have been more convincing if total wall-clock time was measured, since the authors claim that faster training and inference is possible with this method. \n \nTypos: \n1. Algorithm 2: $n$ should be $t$\n2. Critic loss is missing an expectation over data\n3. Line 192: \"we prove that the is *yes*.\"\n\nQUESTIONS:\n1. What is the point of the $Q$ loss $L_{QL}(\\theta)$ in Eq. 9?\n2. Can you provide more analysis of why certain environments didn't benefit from this approach?\n3. What are the actual benefits of this method in terms of total wall-clock time?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SORL, a new offline RL algorithm that leverages expressive generative models as a policy class and allows for direct optimisation of the policy with the gradients of the critic using shortcut models. This avoids the instability of back propagation through time (BPTT) that exists when optimizing vanilla flow/diffusion models with RL due to the iterative denoising process. Using shortcut models allows for using different denoising steps for the policy optimisation and the behaviour cloning terms, achieving the best of both paradigms. It also means that practitioners can scale inference time compute when available, using more steps when precision is important, or fewer steps in compute constrained or latency sensitive environments (like edge deployments). They conduct extensive evaluations to show how SORL achieves better performance compared to existing baselines and also validate their inference time scaling hypothesis. That is, allowing for more denoising steps at inference generally leads to better performance. This is in line with the recent literature studying test time scaling in LLMs and is very useful for practical deployments of offline RL methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Comprehensive evaluation against relevant baselines. The paper presents results on a variety of control tasks and show that SORL beats strong baselines in challenging tasks.\n- The paper validates the claim of inference time scaling, with experiments showing better performance as $M^{inf}$ is increased.\n- Shortcut models preserve the expressivity of the network while also being suitable for RL optimisation without needing a separate distillation step as in FQL.\n- The method is well motivated and easy to implement. \n\nWeaknesses\n- There are some notations and figures which are unclear. See questions below.\n\nOverall I have no major concerns with the paper.\n\nQUESTIONS:\n- I am unsure of the range of $h$ that is used to define the self consistency loss in Eq 5. It seems to be used to both define the number of steps ($\\in [1, M]$)  as well as the dt time interval ($\\in [0, 1]$) that the shortcut model is conditioned on. The text mentions (in L110) that $h \\in [1, M]$ and in L111 $h \\in ( 2^k )_{k=0}^{log_2M}$, where $M$ is the number of discretisation steps. But then shouldn't the $h$ in the flow matching part of $\\mathcal{L}^S$ be 1 and not $1/M$? Since the flow matching loss is for one step. But L112 defined the flow loss as when $h=1/M$ meaning h refers to the dt?\n- In Algorithm 1 is there a reason that you compute the component losses of the actor on separate sections of the batch? That is the Q maximisation loss is only on the first third, the self consistency only on the second third and so on?\n- Also in Algorithm 1, isn't the sign of the grad of the Q term inverted, and it is missing the flow matching term too I think?\n- It might be easier to understand Fig 2 if the x axis was grouped by $M^{BTT}$. That way the gains from increasing $M^{inf}}$ for a fixed training budget (fixed $M^{BTT}$ would be easier to see. Maybe the figure could be split into two, one which tests for scaling inference steps for lower training $M^{BTT}$ and another which tests generalization to inference steps beyond what was used in training.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models to scale both training and inference. SORL's policy can be trained efficiently in a one-stage training procedure. At test time, SORL supports both sequential and parallel inference scaling by using the learned Q-function as a verifier. Experiments are conducted to validate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
**Strengths:**
1. The scaling of the (offline) RL method is an important and interesting problem. 
2. The paper is well written. 

**Weaknesses:**
1. Straightforward Theory. It seems like the conclusion that Theorem 2 tries to prove is trivial: due to the flow matching loss in Eq.2, the policy is learned with regularization, and hence, SORL can learn a policy that is regularized to the behavior of the offline data. It seems like a standard offline RL approach.

2. Lacking Training Scalibility Experiments. The authors claim the proposed method scales both training and inference in the paper. Yet, the training scalability experiments are missing. 

3. Some Equations Flaws. See the question part.

4. Novelty concern. The paper employs an existing type of generative model, the shortcut model (proposed around 2023), to implement the policy in offline RL. Is this enough to construct a novel idea or just a model implementation trick?

QUESTIONS:
Apart from other weaknesses, I list my confusion about the equations in weakness 3 here.

1. In Eq. 7, the minus sign should be plus sign?

2. In Eq. 2, is the trivial solution $f=z_1-z_0$? This makes the objective 0, achieving the smallest value (since $||\cdot||_2>0$).

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to train a shortcut model for offline RL tasks. The training loss consists of three components: flow-matching loss, consistency loss, and Q loss. Unlike two-stage distillation frameworks such as FQL, the proposed method (SORL) is a single-stage approach that offers good scalability at both training and test time.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The question of training one-step generative models directly for rl setting (rather than relying on two-stage frameworks) is timely and important.\n- The method is well designed. By incorporating Q-loss into shortcut model training, it offers scalability in both training and inference.\n- The theoretical analysis appears sound with no obvious flaws. The experiments are standard and show strong performance compared to recent baselines like FQL.\n\n**Weaknesses:**\n\n- Given that scalability is central to the paper, it's worth asking whether the method has been tested on higher-dimensional inputs (e.g., image-based observations). That would further support the scalability claims.\n\nQUESTIONS:\n- **Loss components and data usage (Alg. 1)**: Why does the algorithm use different data samples for computing different loss components? Is there any benefit to this? Would it cause issues if all components used the same batch?\n\n- **Critic update with sampled inference steps**: During critic updates, the algorithm samples $m$ inference steps. When $m$ is small, the resulting actions might be noisy. Could this hurt Q-estimation accuracy and affect Bellman equation convergence—especially if the shortcut model is not yet well trained?\n\n- **Equation 14 and min-Q trick**: I noticed from the appendix that the method uses the min-Q trick. Please consider integrating this directly into Equation 14 for better clarity.\n\n- **Figure 1 (AntMaze-large)**: Why does increasing inference steps lead to worse performance on AntMaze-large tasks?\n\n- **Table 2 (AntMaze-giant)**: SORL performs poorly on AntMaze-giant. It seems that generative RL methods struggle on this task. Any insights into why this might be?\n\n- **Figure 3 (Scalability benefits)**: Does Figure 3 suggest that training-time scalability is more beneficial than test-time scalability? What happens if you increase $M^{\\text{BTT}}$ to 16? Does performance continue to improve?\n\nLIMITATIONS:\nIt would be promising to see the proposed method tested on more high-dimensional offline RL tasks to further validate its scalability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Scalable Offline Reinforcement Learning (SORL), an efficient offline RL algorithm that leverages shortcut models to scale training and inference. SORL incorporates self-consistency into training, enabling it to optimize policies with fewer backpropagation steps while maintaining high expressivity. It achieves strong performance across diverse tasks and can dynamically adjust inference steps based on compute budget, demonstrating improved performance with increased test-time compute.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1.\tThe paper tackles the significant challenge of scaling offline reinforcement learning, which is crucial for practical applications involving large datasets and complex environments.\n2.\tThe authors present their ideas and methods clearly, making the paper accessible and easy to understand, even for those new to the field.\n3.\tThe paper includes a thorough set of experiments across diverse tasks, demonstrating the robustness and effectiveness of the proposed method.\n\nWeaknesses:\n1.\tThe experiments are conducted on a relatively narrow set of environments (8 environments with 40 tasks in total). This might not fully capture the breadth of challenges in offline RL, limiting the generalizability of the results.\n2.\tWhile the paper compares SORL to 10 baselines, the results show that SORL only outperforms them in 5 out of 8 environments. This suggests that its superiority might not be consistent across all types of tasks.\n3.\tThe paper may not provide a comprehensive review of existing methods and their limitations, which is crucial for understanding how SORL fits into the broader context of offline RL research. A more thorough discussion could highlight the unique contributions of SORL more clearly.\n\nQUESTIONS:\n1.\tHow does SORL perform on more complex or diverse tasks beyond the current environments tested? The current results might not reflect its performance on a broader range of real-world scenarios.\n2.\tWhat are the specific factors that contribute to SORL’s superior performance in some environments but not in others? Understanding these factors could provide deeper insights into the algorithm’s strengths and weaknesses.\n3.\tCan the authors provide more detailed analysis or additional experiments to further validate the robustness and generalizability of SORL’s performance? More comprehensive testing might be needed to fully assess its potential.\n4.\tWhile SORL aims to improve training efficiency, what is the trade-off in terms of computational resources required during inference? Does the potential for test-time scaling compensate for any increased computational demands?\n5.\tHow does SORL compare to other recent advances in offline RL, particularly those that also focus on scaling or improving inference efficiency? Are there any key works that should be included in the discussion to provide a more comprehensive comparison?\n6.\tWhat are the potential limitations of using shortcut models in SORL, especially in terms of capturing long-term dependencies or handling highly complex data distributions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}