The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Authors: Jiashun Liu
Johan Obando-Ceron
Pablo Samuel Castro
Aaron Courville
Ling Pan
Venue: ICML 2025 poster (2025)
Keywords: Deep Reinforcement Learning
Sunk Cost Bias
Sample Efficiency
Continious Action Space
Primary area: reinforcement_learning->deep_rl
TL;DR: We observes the sunk cost fallact in Deep RL which may affect the sample efficiency and performance, and propose a simple optimization for unlocking agents' auto-stop ability to mitigate this issue.

Abstract: Off-policy deep reinforcement learning (RL) agents typically leverage replay buffers for reusing past experiences during learning. This can help sample efficiency when the collected data is informative and aligned with the learning objectives; when that is not the case, it has the effect of ``polluting'' the replay buffer with data that can exacerbate optimization challenges in addition to wasting environment interactions due to redundant sampling. We argue that sampling these uninformative and wasteful transitions can be avoided by addressing the **sunk cost fallacy** which, in the context of deep RL, is the tendency towards continuing an episode until termination. To address this, we propose the *learn to stop* (**LEAST**) mechanism which uses statistics based on $Q$-values and gradient to guide early episode termination which helps agents recognize when to terminate unproductive episodes early. We demonstrate that our method improves learning efficiency on a variety of RL algorithms, evaluated on both the MuJoCo and DeepMind Control Suite benchmarks.

The Courage to Stop: in Deep Reinforcement Learning
Overcoming Sunk Cost Fallacy
Jiashun Liu
Johan Obando-Ceron' 23 Pablo Samuel Castro 23
Aaron Courville 23
Ling Pan
Abstract
Avoid pollution
Bad detected, stopl
Off-policy deep reinforcement learning (RL) typi- cally leverages replay buffers for reusing past ex- periences during learning. This can help improve sample efficiency when the collected data is infor- mative and aligned with the learning objectives; when that is not the case, it can have the effect of "polluting" the replay buffer with data which can exacerbate optimization challenges in addition to wasting environment interactions due to wasteful sampling. We argue that sampling these uninfor- mative and wasteful transitions can be avoided by addressing the sunk cost fallacy which, in the con- text of deep RI., is the tendency towards continu- ing an episode until termination. To address this, we propose learn to stop (LEAST), lightweight mechanism that enables strategic early episode ter- mination based Q-value and gradient statistics, which helps agents recognize when to terminate unproductive episodes early. We demonstrate that our method improves learning efficiency on a va- riety of RL algorithms, evaluated on both the Mu- JoCo and DeepMind Control Suite benchmarks.
Replay buffer
Dead
So
1
S2
Current episode
Save Interaction
Reset
Roll out new episode
Environment
Goal
SO
S
1
S2
Figure I. LEAST enables the agent to prematurely end the current episode hy monitoring the quality of present situations, such as getting stuck in suhoptimal trajectory. This mechanism improves sample efficiency, reduces replay buffer contamination, and con- serves the overall interaction budget.
processes. A simple example of this is watching a bad movie until the end because we have already paid for a movie ticket, thereby wasting our time and energy that could have been devoted elsewhere. In this work, we argue that this fallacy is inherent in traditional deep reinforcement learning (deep RL) frameworks, where enforced episode completion hin- ders agents from making strategic stopping decisions (Pardo et al., 2018) 8). It gives the impression that the RL agent is fix- ated on the interaction costs already incurred, which limits its subsequent learning capability and sample efficiency.
"People are reluctant to waste prior investments, even when continuing guarantees further loss."
Halr Arkes
1. Introduction
Specifically, the design limitations of traditional RL, i.e., lacking key mechanisms for autonomous stopping decisions, can limit agents to repeatedly traverse low-quality (sub- optimal) trajectories similar (uninformative) to those they have encountered several times before. Although utilizing experience replay buffers can improve sample efficiency for off-policy deep RL (Fedus al., 2020), the accumu- lation of such experiences, particularly during early train- ing stages, require parameter- -eamettr--ensiiive sampling meth- ods (Schaul, 2015) to prevent overfitting to uninformative experiences (Nikishin et al., 2022).
The sunk cost fallacy refers to a behavioral pattern where agents continue along a current course of action despite recognizing its suboptimality, rather than stop acting if they have already devoted certain amount of effort towards it (Arkes & Blumer, 1985; Sweis et al., 2018). This cog- nitive bias often leads to inefficiencies in decision-making
'Hong Kong University of Science and Technology 'Mila Québec Al Institute 'Université de Montréal. Correspondence to: Ling Pan <lingpan @ hs..
Furthermore, this constrained decision space and restricted decision flexibility fail to optimize the environmental inter- action budget (Yin al., 2023). To address the aforemen-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
tioned fundamental challenge, we introduce Learn to Stop (LEAST), novel approach that empowers RL agents with the ability to terminate episodes prematurely, illustrated in Figure 1.
2. Backgrond
Deep RL The problem of reinforcement learning is com- monly framed within the framework of a Markov decision process (MDP), which is defined by a tuple (S, P, R, 7). Here, S represents the state space, A denotes the action space, P describes the transition dynamics a function S x A x S + [0, 1], R is the reward function defined as S x + R. and 7 (where E [0, 1)) signifies the discount factor.
The agent learns to evaluate the trade-off between continu- ing interaction with the environment and devotes more in- teraction costs and potential benefits considering statistical information of historical experiences. The core of LEAST is an adaptive stopping mechanism that dynamically deter- mines whether to terminate the current trajectory based on its quality and how well we have learned such experiences.
The agent interacts with the environment with its policy T, which maps states actions. The goal is to acquire an optimal policy that maximizes the expected discounted long- term reward. The expected value of a state-action pair (s, a) is defined as Q" a) ET
Specifically, we construct an adaptive stopping threshold by jointly analyzing two key aspects of the agents' past experiences, the central tendency of action values Q and the gradient of critie, which reflect the quality and the learn- ing potential of recent transitions (how well the agent has mastered the current trajectory). separately.
E t = 00 as) 8, ag = a). which means the sum of discounted rewards starting from current transition (s.
In actor-critic methods, the actor Ti and the critic Qo are typically implemented using neural networks, where the parameters of the actor and critic networks are de- noted as Φ and 6, respectively. The critic network is updated by minimizing the temporal difference loss, i.e. (0) = ED [(Qo a) O (s, a))]] where QT(s, denotes the bootstrapping target R(s, a) + 2Q0 + computed using target network parameterized by Φ and 0 based on data sampled from a replay buffer D. The actor network & is typically updated to maximize the Q-function approximation according to VoJ(o) Ep VaQu(s, VoTT(s)
During environment interaction, LEAST compares the pre- dicted Q-value Q against the adaptive threshold. When Q falls below the threshold, it provides an indication that the current partial trajectory's quality and similarity well- learned experiences can be inferior to the historical statistics in memory, triggering strategic early termination of the current episode.
We conduct extensive experiments in standard continuous control MuJoCo tasks and the challenging DeepMind Con- trol Suite benchmarks to demonstrate its effectiveness. On all benchmark tasks, LEAST significantly outperforms the baseline algorithms. Our work opens up a novel direction by teaching agents when to "cut losses"), which is funda- mental capability in human decision- decision-making and represents a novel and orthogonal approach to existing methods for improving efficiency. The main contributions of this paper are summarized as follows:
Sunk Cost Fallacy Sunk cost fallacy (Turpin et al., 2019), also known as sunk cost bias (Arkes & Blumer, 1985), is widely discussed in cognitive science. It describes the ten- dency to follow through with something that people have already invested heavily in (be it time, money, effort, or emotional energy), even when giving up is clearly a better idea. This phenomenon is particularly prevalent in the world of investments (Dijkstra & Hong, 2019). Whether it's in the stock market, real estate, or even a business venture, investors often fall into the trap of holding onto underper- forming assets because they've already sunk so much money into them (Haita-Falle whether deep RI. agents deployed on cumulative reward maximization tasks also suffer from sunk cost fallacy.
We identify critical limitation in current deep RL frame- works where agents continue interacting with the envi- ronment even when encountering familiar low-quality trajectories, leading to wasted environmental interactions and increased sample complexity.
We propose LEAST: a simple yet effective mechanism that enables strategic stopping, significantly improving sample efficiency and performance of existing algorithms.
2017). In this paper, we investigate
We validate the effectiveness of LEAST through compre- hensive experiments across standard benchmarks includ- ing MuloCo (Brockman, 2016) and visual RL scenarios with DeepMind Control (Tassa et al., 2018). Our results demonstrate consistent and significant improvement, il- lustrating LEAST'S efficiency and versatility.
3. Learn to Stop (LEAST)
In this section, we initially investigate whether the sampling process within the current deep RI. algorithm could induce the agent to exhibit signs of sunk cost fallacy and explore the impact of the sunk cost fallacy on learning efficiency, as discussed in $3.1. Subsequently, we introduce a straight- forward and general optimization to the conventional algo-
Please refer Appendix 5 for detailed related work on sample efficiency and early stopping.
2

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
rithmic framework in $3.2, aimed enabling the agent to dynamically overcome sunk cost fallacy, thereby enhancing the learning efficiency of prevalent deep RL methods.
Small
Medium
Large
203
100
20
60
63
3.1. Motivating Example
30
Vanil agent
23
Advoniodd agent
The sunk cost fallacy, prevalent cognitive bias where agents persist with their current course of action consid- ering prior efforts (Dijkstra & Hong, 2019), is inherently present in current deep RL frameworks. Without the deci- sion flexibility to terminate episodes strategically, agents using popular deep RL algorithms, like TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), lack the capability to learn to stop, forcing them to complete entire episodes even when already traversing familiar loa-quality trajectories. It leads to increased interaction costs, and can also result in the replay buffer becoming filled with suboptimal experiences.
Figure 2. Comparison between the vanilla agent and our pro- posed LEAST agent across three PointMaze environments with identical layouts but varying horizon lengths (10, 18, 24). Re- sults averaged over 5 seeds. The Y-axis denotes the normalized score (out of 100), while the X-axis indicates training timesteps. Task details are provided in Appendix A.2.
Finally, we analyze the data distribution within the agents' replay buffer (Figure 3). We partition the data into quadrants based on the mean values of Q (expected reward) and loss (learning signal) from the advanced agent's s buffer samples. The advanced agent significantly reduces the proportion of uninformative samples in the buffer, specifically, those with both low Q-values and low loss (shown in white), which offer little to no effective learning signal. This is achieved by terminating low quality trajectories early. As a result, the buffer contains a higher proportion of high quality data (shown in black), enabling the agent to sample more informative experiences during training and thereby improving learning efficiency.
This phenomenon aligns with findings in value-based deep RL, where using smaller batch sizes has been shown to improve performance by increasing gradient noise, which serves form of implicit regularization and mitigates over- fitting to low-quality transitions in the replay buffer (Ceron et al., 2023). To quantify the impact of this limitation, we conduct controlled experiments to assess the performance disparity between vanilla fuully executed" deep RL agent and an advanced agent, i.e., our method LEAST, equipped with auto-stopping capability (which learns stop upon entering low-quality trajectory that it is already familiar). LEAST is introduced in $3.2.
High
High
We implement both agents based on Soft Cctor-rritic (SAC) (Haarnoja et al., 2018) as the backbone algorithm, and evaluate its learning efficiency on a challenging Point- Maze benchmark as detailed introduced in Appendix A.2 from Hu al. (2023), where there a number of lengthy branches that can trap agents in unproductive trajectories. To ensure fairness, both agents are trained for 106 timesteps with identical hyperparameters and a replay buffer size of 10.. The statistics of the trajectories are visualized in Ap- pendix A.2 which shows that the vanilla agent spends too much time in the "unproductive trajectories".
0.3
0.2
011
HBh
03
0.2
0.1
Hioh
Vanilla Agent
Advanced Agent
Figure 3. Replay buffer data distribution 2 x 10 timesteps. Compared to the vamilla agent, the advanced agent with LEAST sig- nificantly reduces the proportion of low-quality transitions (white region: low Q-value and low learning signal) and increases the density high-quality samples (black region: high Q-value and high gradient magnitude). resulting in more informative and effi- cient training dataset.
Figure 2 demonstrates significantly higher learning effi ciency of the advanced agent with auto-stopping capability compared to the vanilla agent. As the horizon of the maze expands. interaction budgets become more constrained, lead- ing to a correspondingly greater demand for sample effi- ciency. The vanilla agent's performance significantly reduced (maintaining at 60 score), which proves that it is trapped in suboptimal trajectories and unable escape, and the optimization speed is slow due to the decreased sample efficiency. However, the advanced agent nearly maintains a stable performance by avoiding wasteful exploration of known low-quality transitions. This ad advantage becomes tasks.
3.2. Proposed Method
In $3.1, we observed that agents could mitigate the impact of the sunk cost fallacy by stopping and resetting before descending into suboptimal behaviors. In this section, we propose a lightweight general mechanism Learn to Stop
even more pronounced in
3

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
(LEAST) to unlock the agent's auto-stop ability. The high- level idea is to use the recent central tendency of hehavior to construct a sensitivity-awar stop threshold for the current execution. Based on this adaptive threshold, agents can evaluate the risk of falling into suboptimality and decide whether to stop, Empirically, we find that two key factors are essential for the effectiveness of LEAST: (i) identifying the quantifiable and common elemental data to systematically characterize recent policies, and (ii) statistics features of dynamic time scales to sensitively fit on-changing policies.
and brown boxes demonstrates the improved stopping be- havior derived by our method.
While Q-values provide a measure of expected returns for quality, relying solely on them for stopping decisions over- looks a crucial aspect of RL that learns through trial and error: the value of exploration and learning from informa- tive and novel experiences (Pathak et al., 2017; Rashid et al., 2020). Even trajectories with temporarily lower Q-values can provide valuable learning opportunities if they explore unfamiliar state-action regions.
Autonomous Stopping Mechanism The key challenge in designing an effective stopping mechanism is to evaluate the quality and potential learning value of ongoing trajectories during agent-environment interaction, and making timely stopping decisions to save interaction costs to redirect the limited interaction budget to more promising experiences for improving sample efficiency. To address these challenges, we propose dual-criteria approach that dynamically evalu- ates current trajectories from complementary perspectives including the quality and learning potential.
Inspired by Sujit et al. (2023); Terven et al. (2023), we leverage the magnitude of the gradient of the Q-function as a proxy for learning potential, without introducing additional training module and use readily available information from the Q-function.
The intuition is that larger gradients indicate that the agent is encountering novel or poorly understood situations that merit further exploration. We maintain a parallel gradient buffer Bg (with the same structure as BQ), storing the gra- dient magnitudes from recent episodes. For each step i, we compute dynamic weight wi as
We leverage Q-values as the primary quality metric for deter- mining whether an ongoing trajectory is worth continuing.? 2 This choice is motivated by its property that they quantify the expected cumulative rewards for state-action pairs and capture the long-term values of decisions (Van Hasselt et al., 2016). Specifically, we maintain a two-dimensional buffer B3 E IR x that stores Q-values from the K most recent episodes (L indicates the maximum episode length), which provides a dynamic reference for quality assessment. Then, we compare the current Q-value (s;; 0z) at step with threshold derived from historical experience.
Wi Median (Bal:.))
(2)
G2
where Gj is the current gradient magnitude at step i.
To jointly consider quality and learning potential, we modu- late our original quality-based threshold e; with the learning potential weight W; according to W; X c,. Consider the case with a positive 6j, 4; < indicates high exploration and learning potential for unfamiliar states (current gradient G, exceeds the historical median), which leads reduced threshold that encourages continued interaction. On the contrary, wi > 1 will lead to stricter stopping criteria. Our complete stopping criterion in LEAST for each intra-episode step is formalized as follows:
When Q < it indicates that the expected return of the ongoing trajectory falls below the historical average, sug- gesting potentially suboptimal behavior that warrants early termination. However, different stages of an episode have inherently different Q-value scales and behavioral require- ments. We therefore consider step- based independent thresh- olds {e1,,..,+t ...,, for each step instead of a single universal threshold throughout the episode. 3 Considering the noisy distribution of Q-values particularly in the early stages of training, we use the median instead of the arithmetic mean for a more robust threshold caleulation against outliers. For- mally, for each intra-episode step i, the quality-based crite- rion is defined as:
6 2 0: LEAST = Stop & Reset if Q < W/ X E3
6 0: LEAST == Stop & Reset if Q < w, x Ej. (3)
When < O, using w will lead to the opposite effect, for which we switch to -1 Figure 4 shows this dual-criteria threshold (w, x 6, ) significantly outperforms purely Q- based threshold and further improves the performance by effectively trading off quality with potential learning value.
LEAST := Stop & Reset if Q M Median(Bll:d) (1)
Entropy-Aware Dynamic Butfer Size An unstable pol- icy may cause adjacent collected trajectories to vary sig- nificantly, thereby increasing the distributional uncertainty within BQ and making it difficult to estimate its cen- tral tendency. To address this, we compute the entropy H (Bq) = - - P(B2) log P(B₂) every e training steps to assess the orderliness of Bq. and dynamically adjust the size
The effectiveness of the design choice is empirically vali- dated in Figure 4, where the comparison between the orange
2 Note that our method is built upon clipped double Q-learning that mitigates overestimation hias of Q-values.
'We use each minimum value of the subsequent g step-axis after the stop step (min(Bo(:.. g :)) to fill the empty positions.
4

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
error in algorithms makes the idea of accelerating behav- ior change by increasing the UTD affect the performance (Related experiments located in Appendix C.2).
Complete LEAST
LEAST Dynamic buffer
Thus, inspired by Xu et al. (2023), we introduce general episode-tevel schedule for regulating the exploration noise: if agents are forced to stop at an earty stage frequently, which means the policy may stuck in suboptimal land- scape, we will increase the exploration noise a (standard deviation of the Gaussian noise) to help them escape from current behavior. Equation 4 defines how noise is adaptively increased when early stopping occurs frequently:
LEAST w/ Q&Gradlent based threshold
LEAST w/ Q
based threshoid
Vanilla TD3
0.62
0.54
0.66
0.65
0.70
Max Normalized Score
Figure Validation of modules, normalized performance on high-dim task, i.c., large maze (10 seeds). The hox records the final scores of seeds, black line inside the box represents the median, while the horizontal line outside the box records the maximum and minimum values.
max exp( 3
a+), E (0+, (4)
B is the frequency of the most recent m episodes to stop before the e-th step, where ō denotes the upper bound of the noise and a+ is the original setting (lower bond), T; J are the pre-set temperatures to make sure can decay a**. A visualization of the noise schedule is shown in Figure 5.
of the local buffer r. If the current entropy H, exceeds (1+))) x H, where is predefined baseline, we add (or remove) h recent episodes from B accordingly.
Algorithm TD3 with LEAST
Actor: Tg. Double critic networks: Qu Start
0.8
timestep of LEAST: tstart. TD loss: L. Estimated
Q value: Q. Reflection Sets: Bq. Be. Dynamic weight:
0.6
w. Stop threshold: c. Replay buffer: D. pre-set entropy
of Bq: H. Current entropy of Bo: H,. Maximum
0.4
overflow rate: ",, Adjusting scale: h.
0.2
2: while maximum training time do
3: for in range ffizdd episode length) do
0.0
0.2
0.4 0.6
0.8
1.0
4:
4- (8) (+ noise schedule) (Fq
Figure 5. Schedule of exploration noise adjustment. As the fre- quency of premature stopping increases, indicating potential stag- nation in policy improvement, the standard deviation of the ex- ploration noise is gradually increased encourage behavioral diversification and escape from suboptimal trajectories.
5: Observe and new state s'
6:
# Update buffers
7:
Fill D with (s,a, r,
8:
Q1, Q2 < Q0 (8, I < TD error
9:
Fill BQ with min(Q1 (Q1, Q2): Fill BG with L,
10:
EEtropyy -nasepy-based dynamic reflection set
The curve of our complete method in Figure 4 shows that the performance of LEAST becomes more stable with dynamic buffer size and noise schedule.
11:
if H, > >(1 + +) x H then
12:
Add (remove) h episodes + ($3.2)
13:
# Start LEAST
14:
if t 2 Istart then wi - Medson(Balla (Blld) by x Mediann Bo ())
Practical Implementation Pseudocode illustrates the implementation of LEAST integrated with TD3, serving as a representative example how our method can be applied to deterministic policy gradient algorithms.
15:
16:
17:
Stop&Reset or not f- LEAST(W;, 4) (Bq.3}
18:
Update Q F via Fujimoto et al. (2018)
4. Experiments
In this section, conduct comprehensive experiments to evaluate whether LEAST module is necessary for deep RL. We investigate the following key questions: (i($4.1) Can LEAST improve the learning performance of mainstream deep RL algorithms? (ii($4.2)) Is learning to stop also effective in more complex, image-based visual RL tasks?
Adjusting Exploration Noise Empirically, we find that on individual tasks, c.g. Ant, agents have probability of falling into the familiar suboptimal trajectory after stop & re- set due to the unchanged policy, which may limit the benefits of adaptive stopping. However, the cumulative Q estimating
5

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Ant
HaifChoctah
Humanoid
Watker2d
Vinill TD3 TC3 wn LEAST
Step
(a) Verification LEAST built upon TD3
Ant
HalfCheotah
Humanoid
Walker2d
SAC LEAST
Step
Steo
Stop
Step
(b) LEAST built upon SAC
Ant
HalfCheetah
Humanoid
Whlker2d
la REDO
Stop
Seep
5b0p
260
(c) LEAST built upon REDQ
Figure 6. Empirical validation of LEAST across diverse deep RL algorithms in MuJoCo environments. Results show mcan + standard deviation over runs. LEAST consistently improves both sample efficiency and final performance across: (a) TD3, representing deterministic policy methods; (b) SAC, based on maximum entropy reinforcement learning; and (c) REDQ. which employs ensemble of crities for improved value estimation.
iii($4.3)) How sensitive is LEAST to key parameters in each module, as revealed by detailed ablation studies?
celerates carly-stage learning and helps the agent converge to better policies, with particularly notable gains in the Ant and Walker2d tasks. Interestingly, while TD3 with LEAST achieves higher final scores in HalfCheetah, its convergence is slower in the early phase. This may be due to vanilla TD3 prematurely converging to suboptimal strategies. LEAST mitigates this by promoting more effective exploration, en- abling the agent to eventually discover better behaviors in complex decision spaces.
4.1. Improvements for mainstream algorithms
Experimental Setup To verify the generality of LEAST, we select several widely used and mature deep RL algo- rithms as baselines: (i) TD3 (Fujimoto et al., 2018), 1), repre- senting deterministic policy methods; (ii) SAC (Haarnoja et al., 2018), as representative of stochastic policies; (ii) REDQ (Chen et al., 2021), a sample-efficien method that uses an ensemble of critics to improve Q-function fitting: and (iv) DroQ (Hiraoka et al., 2021), a recent variant of SAC that incorporates dropout regularization.
In the case of REDQ (Figure 6(c)), since REDQ already en- hances Q-function fitting via additional critics, its baseline performance is relatively strong. As result, the perfor- mance gains from LEAST are less pronounced during early training. However, LEAST improves sample efficiency dur- ing the middle and later stages and further optimizes final performance.
We evaluate all methods on four core tasks from OpenAl Mujoco-v4, specifically, tasks that SAC and TD3 fail to solve effectively, serving as challenging benchmarks. To ensure fair comparison, we implement all methods using the official codebases and apply consistent hyperparame- ters across experiments. Detailed parameter settings are provided in Appendix B.
Figure 7 provides visual summary of sample efficiency im- provements. The results indicate that LEAST significantly accelerates learning across all three algorithms, with partic- ularly clear gains for TD3 compared to the stochastic-policy baselines. Finally, Figure 8 presents normalized scores across the four tasks to quantify the overall improvements. With LEAST, all algorithms show substantial performance gains: SAC with LEAST reaches performance comparable to DroQ, and REDQ with LEAST achieves the highest over- all scores. However, we observe that LEAST may increase
Results Overall, Figure 6 shows that LEAST significantly improves both learning efficiency and final performance across all three categories of deep RL algorithms, demon- strating its general effectiveness for mainstream deep RL. Specifically, for TD3 and SAC (Figure 6(a, b)), LEAST ac-
6

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
the variance of the algorithms (e-g., wider spread and outliers in the box plots), suggesting potential direction for future work, refining the adaptive evaluation mechanism to enhance training stability.
reference. Hyperparameters for DrQv2 kept consistent across all experiments; see Appendix B for details.
Score
1000000
Vanilla
3800000
11
3600000
Finger Turn Hard
Caripola Swingup
3400000
LEAST
3200000
3000000
2800000
2400000
Hopper Hop
Quadruped Run
2200000
2000000
Figure 9. Performance image-based control tasks. The hor- izontal axis represents training steps; the vertical axis shows the cumulative score. Each curve depicts the mean performance over random seeds, with shaded regions indicating standard deviation
1800000
TD3
SAC
REDQ
Figure Comparison of sample efficiency across MuJoCo tasks. For each algorithm, we report the average number of training steps required to reach the maximum normalized score achieved by its corresponding vanilla (haseline) variant. Lower values indicate faster learning.
AIX
DrQv2
REDQ LEAST
CURL
REDQ
0.55
0.60
0.63
0.75
0.80
Max Normalized Score
DroQ)
Figure 10. Final performance DMC tasks. Results are ag- gregated using the same metric in Figure 4. LEAST performs competitively with feature-based baselines.
SAC w/ LEAST
SAC
TD3 w/ LEAST
Figure 9 shows that although DrQv2 enhances SAC with data augmentation and representation learning, LEAST still provides stable improvements learning efficiency, For ex- ample, in the Finger Turn Hard task, convergence approx- imately 30% faster than vanilla DrQv2. Figure 10 reports normalized scores across all tasks. While CURL and A- LIX improve feature representations to accelerate learning, LEAST outperforms both and approaches the performance of TACO. These results suggest that in visual RL, unlock- ing adaptive stopping can be a promising and cost-effective alternative to heavy architectural modifications, requiring additional networks improve sample efficiency.
TD3
0.72 0.74
Max Normalized Score
Figure Final performance MuJoCo task, aggregated using the same evaluation metrics as in Figure 4. Results are normalized averaged over random seeds. Higher values indicate better overall task performance.
4.2. Effectiveness on Visual Reinforcement Learning
Experimental Setup To evaluate the generalization of LEAST to the visual domain, we adopt DrQv2 (Yarats et al., 2021), ofate-of-the-art Vision RL algorithm, as our back- bone. We select four image-based control tasks from the DeepMind Control Suite (Tassa et al., 2018) examine whether the benefits of LEAST transfer visual RL set- tings. Additionally, we include several DrQv2-based base- lines that introduce observation encoders: CURL (Laskin et al., 2020), A-LIX (Cetin et al., 2022), and TACO (Zheng et al., 2024). Among these, TACO represents a recently proposed variant of DrQv2, which we use as the SOTA
4.3. Robustness of Median Threshold Estimation
As analyzed in $3.2, we evaluate the individual contributions of LEAST components. Here, compare the use of the median the arithmetic mean estimate the central tendency of BQ on the Ant task. Figure 11 shows that the median yields a more stable estimate, avoiding spikes. This robustness arises because the median is less sensitive to outliers from recent trajectories, thereby mitigating policy collapse caused by frequent early stopping.

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Task
0.15
0.25
0.35
0.45
820
LEAST w/ average
TD3 0.684 0.026
0.695
0.035
0.694
0.028
0.671 +: 0.049
720
LEAST w/ median
SAC
0.697 0.032
0.706
0.039
0.688
0.057
0.663 = 0.074
520
sco
Table I. Ablation study of a. Values average over 5 random seeds.
320
Task
TD3 0.681 0.062
0.685 0.028
0.696
0.037
0.679 0.053
Step
Figure 11. Comparison of methods for estimating the central tendency of Big on the Ant task. Using the median provides more stable and robust estimates than the arithmetic mean, particularly in the presence of outliers. This stability helps mitigate overesti- mation in early stopping decisions and improves the reliability of the adaptive threshold used in LEAST.
SAC 0.692 0.038
0.706 0.039
1.696
0.031
0.701 + 0.036
Table 2. Ablation study of e. Values average over 5 random seeds.
4.6. Impact of Entropy-Guided Buffer Resizing
This section evaluates the sensitivity of LEAST to the catropy-based dynamic scaling parameter y. We test this using TD3 and DrQv2 across all scenarios. Table 3 shows that y E [0, 0.1] yields robust performance. We attribute this the instability of early-stage trajectories, where larger Y may mischaracterize the behavioral distribution due to noise.
4.4. Sensitivity to the Learning Potential Weight (w)
We now examine the role of the learning potential weight in LEAST, which modulates the Q-value threshold to balance trajectory quality with potential learning value. Fig- ure 12 illustrates the effect of varying the importance weight in the threshold calculation. Compared to TD3, SAC more sensitive to this parameter. A value in the range (0.3, 0.6] provides a robust trade-off between and w for both algorithms. This highlights the importance of balanc- ing quality- qualety-based based stopping with learning potential signals when designing adaptive termination criteria.
0,05
0.15
0.20
DMC 0.724 + 0.047
0.743 + 0.062
0.708
0.076
0.712
0.006
0.715 + 0.022
MilaCo 0.083 0.043
0.706 + area
0.703
0.94
0,702 + 0.021
0.688 + 0.064
Table 3. Ablation study of yy Values average over 5 random seeds.
4.7. Influence of Start Time and Initial Set Size
0.95
Table 4 analyzes the impact of the LEAST start time. For MuJoCo tasks, initiating at 10-20% of training is optimal, while visual tasks (e.g.. DMC) benefit from earlier starts (5-15%). Table 5 investigates the effect of the initial set size for BQ- We recommend using 250 episodes. Too small set may lead to poor statistical estimates, while too large may over-smooth recent behavior.
Normalized score
0.90
0.85
TD3 w/ LP
0.80
SAC w/ LP
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Tusk
0.25M
0.5M
0.75M
IM
Figure/ 12. Sensitivity analysis of the weighting factor w in the adaptive stopping threshold of LEAST. We vary the scaling coefficient applied to w across a range of values and evaluate its impact on final performance. Results are averaged over 5 seeds. This analysis reveals that LEAST is robust to moderate changes in w. with the best performance typically achieved when w lies in the range [0.3, 0.6].
5492,48
018.71
6547.57
6408 246.33
Quadruped run
947,
937.
54.75
915.02
Table 4. Ablation study of start time. Values average over random seeds.
Am Quadraped nu
4718.47 1009 750.33 a 96.53
726.05
23
6792 45 327.00 9b., 00 30.49
941.37 25.35
963 25
26
Table 5. Ablation study of initial set size. Values average over 5 random seeds.
4.5. Effecc of Adaptive Noise Scaling
This section analyzes the sensitivity of our method to param- eters in the noise schedule. Table shows that the optimal value of o lies within [0.25, 0.35] for TD3 and [0.15, 0.25] for SAC. We hypothesize that SAC, being stochastic pol- icy, inherently exhibits behavioral diversity and thus benefits less from large noise. Table 2 evaluates the length threshold e. For MuJoCo, a range of [400, 500] is empirically robust with m = 50 episodes used for statistical estimation.
5. Related Work
Sample Efficiency in Deep RL Clipped Double Q- learning (Fujimoto et al., 2018) is a landmark approach that introduces a double Q network to mitigate overestima- tion in reinforcement learning (Hasselt, 2010; Van Hasselt
8

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
al., 2016; Song et al., 2019; Pan et al., 2019; 2020; 2021). Building on this, Moskovitz al. (2021) proposed an online method to tune pessimism and correct for the underestima- tion introduced by TD3. Nauman et al. (2024) demonstrated that using a residual network backbone can further alleviate pessimistic Q-learning. Chen et al. (2021) addressed esti- mation bias by increasing the number of Q networks to an ensemble of ten. Other studies have shown that scaling up DQN architectures (Schwarzer et al., 2023; Obando-Ceron et al., 2024b;a; Sokar et al., 2025) and increasing the replay ratio (D*Oro et al., 2022) can also accelerate Q-function learning.
decisions in complex scenarios, e-g., optimizing the combi- nation of Q-value and loss to more accurately characterize trajectory quality. (ii) As shown in Figure 3, the proportion of samples with high loss but low Q-value (blue region) remains largely unchanged. These are likely low-quality transitions with minimal expected return. Future work could investigate refining the sampling threshold by incorporating additional indicators to better filter out such data. (iii) After reset, agents may re-enter previously suboptimal trajectories. Introducing a noise-based diversification module could help, though care must be taken to properly schedule the noise magnitude and timing to balance exploration with training stability.
Another line of work focuses on optimistic exploration strategies (Wang et al., 2019; Moskovitz et al., 2021). For instance, ICM (Pathak et al., 2017) uses forward prediction error as curiosity signal to encourage exploration of novel states. Recent research also advocates the use of generative models such as VAEs (Imre, 2021), GANs (Huang et al., 2017; Daniels et al., 2022), and diffusion models (Lu et al., 2024; Yu et al., 2023) to augment data and meet sampling requirements. However, existing methods largely overlook the impact of sunk cost fallacy in deep RL. This paper aims to fill that gap by proposing adaptive stop sampling as a cost- effective strategy to improve the performance of deep RL algorithms, direction that has been largely underexplored by the community.
To ensure consistency and reduce computational demands, we adopted the default hyperparameters provided by each baseline model for all experiments. While this choice facil- itates reproducibility, it is well known that RL algorithms can exhibit significant sensitivity to hyperparameter settings (Ceron et al., 2024; Ceron & Castro, 2021). A thorough hyperparameter sweep for each experimental configuration would be preferable principle, but remains impractical given the associated computational expense
7. Conclusion
In this paper, we highlight the presence of sunk cost fal- lacy in mainstream deep RI. algorithms. This fallacy leads agents to persist in executing entire episodes, even when trajectories become suboptimal, resulting in unnecessary interactions and buffer contamination, ultimately hindering learning. We identify this as a previously overlooked lim- itation in the literature: the inability of current algorithms to terminate bad trajectories early may significantly impact overall performance. To address this, we propose LEAST, a direct optimization approach that quantifies trajectory qual- ity and enables early stopping, without requiring additional network components. LEAST effectively mitigates the sunk cost fallacy and improves learning efficiency across vari- ety of tasks. We hope that our findings will inspire further research aimed at optimizing sampling and learning effi- ciency.
Early Stopping in Deep RL To the best of our knowl- edge, there is limited work on adaptive termination in online deep RL. Existing studics primarily examine performance in environments with uncertain episode lengths or time limits (Pardo et al., 2018). Poiani et al. (2023; 2024) theoretically demonstrate that truncating trajectories based on prior perience can accelerate learning in Monte Carlo methods. Mandal et al. (2023) show that policies can benefit from episode lengths that increase linearly in extended-duration tasks. In the context of Constrained Markov Decision Pro- cesses (CMDPs), some works have explored early termina- tion based on prior safety signals or contextual knowledge (Sun al., 2021), as well as using offline data to estimate reward distributions (Killian et al., 2023), with applications in domains such as healthcare (Fatemi et al., 2021). These insights on how trajectory length influences policy learning provide useful foundations and motivation for our research.
A Acknowledgment
This work is supported by the National Natural Science Foundation of China 62406266. The authors would like to thank Roger Creus and Gopeshh Subbaraj for valuable dis- cussions during the preparation of this work. We thank the anonymous reviewers for their valuable help in improving our manuscript. We would also like to thank the Python com- munity (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing tools that enabled this work, including NumPy (Harris et al., 2020) and Matplotlib (Hunter, 2007).
6. Future Work and Limitations
We encourage the community to further investigate the sunk cost bias in reinforcement learning. Based on our find- ings, we identify three promising directions for future work: (i) While LEAST improves performance, stability can still be enhanced. Future research could focus on developing more sensitive and robust evaluators to improve stopping
9

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Impact Statement
Fatemi, M., Killian, T. W., Subramanian, J., and Ghassemi,
M. Medical dead-ends and learning to identify high-risk states and treatments. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4856-4870. Curran Associates, Inc., 2021. URL httpss//proceedingg neurips.. cc cc/paper /c//apee_iieee/ppe/////////// 2 4400199cc1aaP0133356444477438813 pdf.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp. 620-636. PMLR, 2023.
Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisit- ing fundamentals of experience replay. In International conference on machine learning, pp. 3061- 1061-3071. PMLR, 2020.
Arkes, H. R. and Blumer, C. The psychology of sunk cost. Organizational behavior and human decision processes, 35(1):124-140, 1985.
Brockman, G. Openai gym. 100,000006.1140. 2016.
arXiv preprint
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference on machine learning. pp. 1587-1596. PMLR, 2018
Ceron. J. O. and Castro, P. S. Revisiting rainbow: Pro- moting more insightful and inclusive deep reinforcement learning research. In International Conference on Ma- chine Learning, pp. 1373-1383. PMLR, 2021.
Haarojja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Ceron, S. O., Bellemare, M. G., and Castro, P. Small batch deep reinforcement learning. In NeurIPS, 2023.
Ceron, J. S. O., Araújo, G. M., Courville, A., and Castro, P. On the consistency of rrper-parameter selection in value-based deep reinforcement learning. In Reinforce- ment Learning Conference, 2024.
Haita-Falah, C. Sunk-cost fallacy and cognitive abil- ity in individual decision-making. Journal of Eco- nomic Psychology, 58:44 59, 2017. ISSN 0167- 4870. doi: https:/dbb.orr//0.0011jj...ppppp
001. URL UtL.itppss/www.ssseeeeeeeee..... scteanee/arrtacee////////7777700
Cetin, E., Ball, P. J., Roberts, S., and Celiktutan, O. Stabi- lizing off-policy deep reinforcement learning from pixels. arXiv preprint ar arXiv:2207.00988. 2022.
Harris, C. R., Millman, J., Van Der Walt, S. Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy, Nature, 585(7825)335-362, 2020.
Chen, X., Wang, C., Zhou, Z., and Ross, K. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint 10X.:22001000900 2021.
Daniels, 7., Raghavan, A., Hostetler, J., Rahman, A., Sur, I., Piacentino, M., and Divakaran, A. Model-free generative replay for lifelong reinforcement learning: Application to starcraft-2, arXiv preprint arXiv:2208.088 050056. 2022.
Hasselt, H. Double q-learning. Advances in neural informa- tion processing systems, 23, 2010.
Hiraoka, T., Imagawa, T., Hashimoto, T., Onishi, T., and Tsuruoka, Y. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:21 2021.
Dijkstra, K. A. and Hong, Y-y. The feeling of throw- ing good money after bad: The role of affective re- action in the sunk-cost fallacy. PLOS ONE, 14(1):1- 19, 01 2019. URL /ectps:///oi.. http///i..rrg//.1111111 journal pone. 0209900.
Hu, E. S., Chang, R., Rybkin, O., and Jayaraman, D. Planning goals for exploration. arXiv preprint arXiv:2303.1300 2023.
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-I.., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NewrIPS 2022, 2022.
Huang, V., Ley, T., Michou-Konchylats M., and Hu, W. Enhanced experience replay generation for efficient re- inforcement learning. arXiv preprint arXiv: 2017.
10

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Hunter, J. D. Matplotlib: A 2d graphies environment. Com- puting in science engineering, 903))90-99) 2007.
Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 1ei1-122891883,,, 2021.
Imre, B. An investigation of generative replay in deep reinforcement learning. B.S. thesis, University Twente, 2021.
Nauman, M., Ostaszewski, M., Jankowski, K., Mitos, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16155 2024.
Killian, T. W., Parbhoo, S., and Ghassemi, M. Risk sensitive dead-end identification in safety-criticn offline reinforce- ment learning. arXiv preprint arXiv:2301 1.56646644 2023.
Laskin, M., Srinivas, A., and Abbeel, P. CURL: Con- trastive unsupervised representations for reinforcement learning. III, H. D. and Singh, A. (eds.), Pro- ceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5639-5650. PMLR, 13-18 Jul 2020. URL Stthtt::: / http:///ooeeedeeeeeeeeeee mlr aalr.ppee press// v11 /tw.///annn//// html.
Nikishin, E., Schwarzer, M., D' D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias deep reinforcement learning. In International conference on machine learn- ing. pp. 16828-16847. PMLR, 2022.
Obando-Ceron, J., Courville, A., and Castro, P. S. In value- based deep reinforcement learning, a pruned network is a good network. Architecture, 4:4-5, 2024a.
Li, B., Tang, H., Zheng. Y., Hao, J., Li, P., Wang, 7.,, Meng, Z., and Wang, L. Hyar: Addressing discrete-contimuuus action reinforcement learning via hybrid action represen- tation. arXiv preprint 2Xiv:2/09.05490 2021.
Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J., Dziugaite, K., Precup, D., and Castro, Mixtures of experts unlock parameter scaling for deep rl. In Proceedings of the s1 International Conference on Machine Learning, pp.
2024b.
Lillicrap, T. Continuous control with deep reinforcement learning, arXiv preprint 0ttp::150902277..1 2015.
Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 93)):0--00, 2007. doi: 10. 1109/MCSE.2007.588
Lillicrap. T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning, arXiv preprint arXiv: 02971, 2015.
Pan, L., Cai. Q., Meng, Q., Chen, W., Huang, L., and Liu, T.-Y. Reinforcement learning with dynamic boltzmann softmax updates. arXiv preprint 1903.v1903.0599 2019.
Liu, J., Jianye, H., Hao, X.. Ma, Y., ZHENG, Y., Hu, Y., and Lv, T. Unlock the intermittent control ability of model free reinforcement learning. The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Pan, L., Cai, Q., and Huang, L. Softmax deep double deter- ministic policy gradients. Advances in neural information processing systems, 33:1 1767-117777 2020.
Liu, J., Ceron, J. S. O., Courville, A., and Pan, L. Neu- roplastic expansion in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /pppneevee.....eee net/ forum? 2d=20g22227aa..
Pan, L.,, Rashid, T., Peng. B., Huang, L., and Whiteson, Regularized softmax deep multi-agent q-learning. Ad- vances in Neural Information Processing Systems, 34: 1365-1377, 2021.
Pardo, F., Tavakoli, A., Levdik, V., and Kormushev, Time limits in reinforcement learning. In International Con- ference Machine Learning, pp. 4045-4054, PMLR, 2018.
Lu, C., Ball, P., Teh, Y. W., and Parker-Holder, J. Syn- thetic experience replay. Advances Neural Information Processing Systems, 36, 2024.
Ma, G., Li, I.., Zhang, S., Liu, 7.,, Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint arXiv:2310.074/66 2023.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell. T. auriosity-driven exploration by self-supervised predic- tion. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Mandal, D., Radanovic, G., Gan, J., Singla, A., and Ma- jumdar, R. Online reinforcement learning with uncertain episode lengths. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9064-9071, 2023.
Poiani, R., Metelli, A. M., and Restelli, M. Truncating trajectories monte carlo reinforcement learning. In In- ternational Conference on Machine Learning, pp. 27994 28042, PMLR, 2023.

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Poiani, R., Nobili, N., Metelli, A. M., and Restelli, M. Trun- cating trajectories in monte carlo policy evaluation: An adaptive approach. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Turpin, M. H., Meyers, E. A., Fugelsang, J. A., Friedman, O., and Biatek, M. Sunk cost bias and withdrawal aver- sion. The American Journal of Bioethics, 19(3):57-59, 2019.
Rashid, T., Peng, B., Boehmer, W., and Whiteson, S. Opti- mistic exploration even with pessimistic initialisation. arXiv preprint arXiv:2002.12174. 2020.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-leamnig. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Schaul, Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
Van Rossum, G. and Drake Jr, L. Python reference man- ual. Centrum voor Wiskunde Informatica Amsterdam, 1995.
Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
Wang, Y., Wang. R., Du, S. S., and Krishnamurthy, A. Optimism in reinforcement learning with gener- alized linear function approximation. arXiv preprint 1rXiv:192.0044 2019.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon deep reinforcement learning. In International Conference on Machine Learning, pp-
Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji. T.. Luo, Y., Liu, X., Yuan, J., Hua, P., Li, S., Ze. Y., Daum'c e. H., Huang, F., and Xu, H. Drm: Mastering visual reinforcement learning through dor- mant ratio minimization. ArXiv, 0os/2310.1966. 2023. URL https: /appi/app.a/a//// wwwanti org Co:ppsII::24777770
32145-32168. PMLR, 2023.
Sokar, G., Obando-Ceron, J., Courville, A., Larochelle, H., and Castro, S. Don't 't flatten, tokenize! unlocking the key to softmoe's efficacy deep rl, 2025. URL https:/ / arxiv.. org/ abs 2200 01930.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented re- inforcement learning. arXiv preprint arXiv:2107.09645. 2021.
Song. Z.. Parr, R., and Carin, L. Revisiting the softmax bellman operator: New benefits and new perspective. In International conference on machine learning, pp. 5916- 5925. PMLR, 2019.
Yin, D., Thiagarajan, S., Lazie, N., Rajaraman, N., Hao, B., and Szepesvari, C. Sample efficient deep rein- forcement learning via local planning. arXiv preprint arXiv:2301. 12579, 2023.
Sujit, S., Nath, S., Braga, P., and Ebrahimi Kahou, S. Priori- tizing samples reinforcement learning with reducible loss. Advances in Neural Information Processing Systems, 2023.
Yu, T., Xiao, T., Stone, A., Tompson, J., Brohan, A., Wang, S., Singh, J., Tan, C., Peralta, J., Ichter, B., et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. 1550, 2023.
Sun, H., Xu, Z., Fang, M., Peng, Z., Guo, J., Dai, B., and Zhou, B. Safe exploration by solving early terminated mdp. arXiv preprint arXiv:2107.04200, 2021.
Sweis, B. M., Abram, S. V., Schmidt, B. J., Seeland, K. D., MacDonald, A. W., Thomas, M. J., and Redish, A. D. Sensitivity to "sunk costs" in mice, rats, and humans. Science, (6398))))178-11 2018, doi: 10. 10.1s26/scee aar8644. URL httLhttp::/// scwencee orgoorrgoooo. doi/ abs/10 112111222 20111///cc/ wtt.111////eeeeaaaa.
Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., Daumé III. H., and Huang, F. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., al. Deepmind control suite. arXiv preprint 2018.
Terven, J., Cordova-Esparza, D. M., Ramirez-Pedraza A., Chavez-Urbiola E. A., and Romero- Homero-Gonzalez, J. A. Loss functions and metrics in deep learning. arXiv preprint arXiv:2307.02694 2023.
12

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
A. Related Preliminaries
A.I. Baselines
TD3 In our paper, we utilize TD3 as the representative of a deterministic policies. TD3, an Actor- algorithm, is widely used in various decision scenarios as the baseline (Li et al., 2021; Liu et al.), and has derived a wide range of variants to establish new SOTA many times. Differs from the traditional policy gradient method, DDPG (Lillicrap et al.. 2015), TD3 employs two heterogeneous critic networks, denoted as Q0 mitigate the issue of over-optimization in Q-learning. Thus, the Loss function of critics is:
(0)) E [(v (8, for Vi E {1,2).
(5)
Where y = r + 7 min Q5 m)("))) o denotes the target network parameters. The actor is updated according to the Deterministic Policy Gradient (Fujimoto et al., 2018):
j=1.2
V J(b E. V (8, (s, ())(())
(6)
SAC We choose SAC (Haarnoja et al., 2018) as a representative of the stochastic policy in combination with LEAST in the main experiment. SAC is designed to maximize expected cumulative rewards while also promoting exploration through the principle of maximum entropy. The actor aims to learn a stochastic policy that outputs a distribution over actions, while the critics estimate the value of taking a particular action in a given state. This allows for a more diverse set of actions, facilitating better exploration of the action space. In traditional reinforcement learning, the objective is to maximize the expected return. However, SAC introduces an additional term that maximizes the entropy of the policy, encouraging exploration. The goal is to optimize the following objective:
(TT) = a,,,, [r(s,, a.) + aHd (x((+s))
(T((1s)) is the entropy of the policy. a is a temperature parameter that balances the trade-off between the reward and the entropy. The training process for SAC consists two main updates: updating the value function and updating the policy. The value function updated by minimizing the following loss:
L(Q) = (ss...rs))) 1
(00:0) -r
(r+
y is the discount factor, dictating how much future rewards are valued. V(s') is the value function of the next state, typically approximated using separate neural network. The policy is updated by maximizing the following objective:
((x) = EB, [Q(se, ax) - a log ([ee(s))]]
-a (aa) (s:) represents the entropy of the policy, promoting exploration.
REDQ Chen et al. (2021) found that the accumulation of the learned Q functions' estimation bias over multiple update steps in SAC can destabilize learning. To remedy this bias, they increase the number of Q networks from to an ensemble of 10. Its training procedure is consistent with SAC. REDQ is recognized as an effective algorithm to improve the sampling efficiency of deep RL.
DrQv2 DrQv2 is model-free off-policy algorithm for image-based continuous control. DrQ-v2, an actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements including: switch the base RL learner from SAC to DDPG, Incorporating n-step returns to estimate TD error. They also find useful to apply bilinear interpolation on top of the shifted image for the random image augmentation and to optimize the exploration schedule.
A-LIX & TACO
In this paper we chose two well-known variants of DrQv2 as the baseline to evaluate the research value of adaptive stopping as a new research line. A-LIX try mitigate catastrophic self-overfitting in DrQv2 from the feature representation level, that is, providing adaptive regularization to the encoder's gradients to stabilize temporal difference learning from encoders. And it improves the performance of DrQv2 on diverse image input tasks.
TACO also optimize DrQv2 by adding an additional feature extraction model. Practivally, TACO incorporates an auxiliary temporal action-driver contrastive learning objective to learn state and action representations through mutual information, and pelf-prediction representations. It established new SOTA on DMC and metaworld benchmarks.
13

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
A.2. Maze
We the widely used open-source Point Maze scenario build the three tasks of small, medium and large. The specific layout is shown in Figure 13.
Target
area
Agent
Figure 13. Layout of the maze.
The agent, 2D point, is noisily initialized in the bottom left. test time, we evaluate on reaching the right corner. The 2-D state and action space correspond to the planar position and velocity, respectively. The episode length 50 steps. The top right corner is the hardest to reach, so we set this to be the test-time goal. We increase the point size 1.25x, with speed factor of 0.9. The dense reward is designed according to the L2 distance from the center of the target region to the point. The parameters for SAC are consistent with the recommendations, with set to 0.95.
We further count the final positions of the last 50 episodes of each run for the vanilla agent as well as the advanced agent (LEAST) in Figure 14. The results show that the advanced agent can effectively avoid multiple dead ends and successfully reach the target area under the limited interaction budget.
A.3. Measurement
Fraction of Active Units (FAU) Although the complete mechanisms underlying plasticity loss remain unclear, a reduction in the number of active units within the network has been identified key factor contributing to this deterioration in both visual RL (Ma et al., 2023; Sokar et al., 2023) and traditional RL (Abbas et al., 2023; Liu et al., 2025). Consequently, the fraction of active units (FAU) is widely utilized a metric for assessing plasticity. Specifically, the FAU for neurons located in layer N, denoted as VVV is formally defined as:
R/V EnEN ((an(z) 0)
where a, (2:) represent the activation of neuron given the input x, and is the neuron within layer N. The meaning of this metric is the proportion of the number of activated parameters to the total number of parameters. That is to say, the slower this ratio decreases, the slower the plasticity loss of the network is, so as to maintain the representation ability the quantity network for new data. This allows it to be visually visualized following the training time steps, which can be used evaluate the continuous learning ability of deep RL. agents.
14

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
(a) Vanilla Agent
(b) Advanced Agent
Figure 14. Statistics of the final positions of two agents on the medium task.
A.4. Sunk Cost Fallacy
Sunk cost fallacy (Turpin et al., 2019), also known as sunk cost bias (Arkes Blumer, 1985), is widely discussed in cognitive science. describes the tendency to follow through with something that people have already invested heavily in (be it time, money, effort, or emotional energy), even when giving up clearly better idea. This phenomenon is particularly prevalent in the world of investments (Dijkstra Hong, 2019). Whether it's in the stock market, real estate, business venture, investors often fall into the trap of holding onto underperforming assets because they've already sunk much money into them (Haita-Fale 2017). In this paper, investigate whether deep RL agents deployed on cumulative reward maximization tasks also suffer from sunk cost fallacy.
B. Experimental Details
B.1. Structure
TD3 In this paper, we use the official architecture, and the detailed structure shown in Tab 6.
Layer
Actor Network
Critic Network
Fully Connected
(state dim, 256)
(state dim, 256)
Activation
ReLU (256. 128)
ReLU
Fully Connected
(256, 128)
Activation
ReLU
ReL.U
Fully Connected
(128, action dim) and (128, I)
Activation
Tanh
None
Table Network Structures for TD3
SAC & REDQ In this paper, we follow the setting that commonly used in Dep RL community, and the detailed structure is shown Tab 7. We set the number of ensembles to 10 according to the official setting of REDQ ht : / /gtbbbb....tg//g///g com/ Sy571/Randomi Aadd-Ennebbb.. eedeedd-oieo Double- learning-n---eeeee RED-n-------------- HEED--/blll /blob/ /main, but for fairness, the UTD of all algorithms is set 1.
15

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Layer
Actor Network
Critic Network
Fully Connected
(state dim, 256)
(state dim, 256)
Activation
ReLU
ReLU
Fully Connected
(256, 128)
(256, 128)
Activation
ReLU
ReLU
Fully Connected
(128. 2x action dim) and (128. 1)
Activation
Tanh
None
Table 7. Network Structures for SAC
DrQv2& TACO& A-LIX We use these three methods to conduct experiments on robot control tasks within DeepMind Control using image input as the observation. All experiments are based on official codes. Code of DrQv2 can be found at https: //github. www.//o////aaae http//taabbiiiigiieeeeeaeeeeeete http:////s..aaaaaaaoooooo---ooo-
the website of TACO is https: http.///raaaacceegg222///OO and A-LIX code can be downloaded at https: httpp::/gittiii /tt/t/gghttbb// com/ masstr.. We maintain the unchanged architecture from the
http:///tttu..ccooccoeeeeeeeeraa com/ wawcebookreeercc//ttro....
official setting.
B.2. Implementation Details
Our code implemented with Python 3.8 and Torch 1.12.1. All experiments were run on NVIDIA GeForce GTX 3090 GPUs. Each single training trial ranges from 10 hours to 21 hours, depending the algorithms and environments, e.g. DrQv2 spends more time than TD3 to handle the image input and DMC needs more time than OpenAl mujoco for image rendering.
TD3 Our TD3 is implemented with reference to gi github. com commsfujiimTI (TD3 source-code). The hyper-parameters
for TD3 are presented in Table 8. Notably, for all OpenAl MuloCo experiments, we use the raw state and reward from the environment, and no normalization or scaling is used. An exploration noise sampled from N(0, 0.1) (Lillicrap, 2015)
added to all baseline methods when selecting an action. The discounted factor is 0.99 and we use Adam Optimizer (?) for algorithms. Tab.8 shows the hyperparameters of TD3 used in all our experiments. We suggest using seed - 7 reproduce the learning curve in the main text.
Hyperparameter
TD3
TD3 w/ LEAST
Actor Learning Rate
le-4
le-4
Critic Learning Rate
le-3
le-3
Discount Factor
0.99
0.99
Batch Size
128
128
Buffer Size
1e6
1e6
Table Shared hyperparameter setting of TD3.
SAC Our TD3 is implemented based on https: http://tttbbb.ccc/gg/////2222/2/ reliable open source implement in Pytorch style). The hyper-parameters for SAC are recorded Table
a
DrQv2 We use DrQv2 as the backbone to verify our methods on DeepMind Control tasks. The details can be seen in Tab. 10. We follow the setting used in DrQv2, for Walker Stand/Walk/Run tasks, we set the mini-batch size as 512 and n-step return of 1. In the Quadruped Run task, we use replay buffer of size 10 and set the learning rate to 8 10-5 We also increase the feature dimensionality to 100 in Humanoid scenarios.
Lern to Stop For our Learn stop schedule in TD3, we generally set the start time 0.6. step for most OpenAl MuJoCo tasks, expect Ant (start at 0.75.M)and HalfCheetah (start at 0.25M). For our Learn to stop schedule in SAC (REDQ), we set the start time as 0.5M step for most OpenAI MuJoCo tasks, expect Ant (start at 0.65.M Jand HalfCheetah (start at 0.25.M). And for DMC tasks, the schedule will be turn on at 0.4M step. As for the initial size of the reflection sets, we set 150 for both TD3 and SAC for OpenAl muJoCo tasks, and 250 for image input tasks. The scale of Loss ratio w is set as 0.5 for all tasks, but 0.7 for Humanoid and 0.6 for HalfCheetah in SAC. The max threshold the max Q value in BQ
16

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Hyperparameter
SAC
SAC w/ LEAST
Optimizer
Adam
Adam
Learning
10
10
Discount factor
0.99
0.99
Number of hidden layers
2
Number of hidden units per layer
256
256
Activation Function
ReLU
ReLU
Batch size
256
256
Warmup steps
5000
5000
Target update
0.005
Target update interval
Actor update interval
Eintppy target
-|4|
-|4]
Table 9, Shared hyperparameter setting of SAC.
Hyperparameter
DrQ
DiQ w/ LEAST
Replay buffer capacity
Action repeat
2
2
Seed frames
4000
4000
Exploration stops
2000
2000
n-step
3
Mini butch size
256
256
Discount y
0.99
0.99
Optimizer
Adam
Adam
Learning rate
le
1e
Agent update frequency
2
Critic Q function sofl- update rale
0.01
0.01
Features dim.
50
50
Hidden dim.
1024
1024
Exploration stddev. clip Exploration strdle. schedule
0.3
(1.0001 500000), Humanoid tsk: 0, 2000000)00000))000000) [10,00(1,0001,, 5000000, Humanoid tsks: 1inear(1.0, (1.0,.1 2000000)
Table 10. A default set hyper-parameters used our DrQv2-hased methods.
(we find that 0.75#median Q can aslo be magic number) and the min threshold is set min value in BQ (0.25x median Q is a also good choice). Too many sensitive parameters are also an issue to be optimized in the future of this work. To prevent extreme threshold adjustments, we use Clip function to limit wi x c E [Qmin Qi)z where Qmaz [t] denotes the highest Q value among intra-episode step Qmin(]) [X] the min value.
C. Additional Experiments
C.1. Effects on Continuous Learning and Plasticity
Nikishin et al. (2022) finds that the rapid loss of plasticity in network caused by overfitting to noisy data in the early training is key reason for losing continuous learning ability. In $3.2, we observe that LEAST can effectively optimize the data distribution in the buffer, thus avoiding the agent from training via useless (noisy) transitions too often. This may go some way to overcoming aformentioned issue and maintaining the continuous learning ability (mitigating the loss of network plasticity).
In this section, we verify in detail whether the above advantage exists in our method. We conduct a evaluation on the difficult long-time time learning task humanoid walk, i.e. DrQv2 needs about 6x steps than normal DMC tasks to learn sequential of skills (15M). The parameters remain the same in $4.2. Notably, we introduce widely used metric,the Fraction of Active Units (FAU*), to quantify the plasticity of the network. FAU is used in such a way that the gentler the downward trend of the curve indicates the metter-maintaing plasticity of the network. Figure 15 shows that our method decreases FAU slower than the baseline and achieves impressive performance on long-term learning tasks. This corroborates the positive effect of LEAST on maintaining the continuous learning ability. In the future, we will deeply analyze the theoretical correlation between plasticity and adaptive stopping.
Plastticity measurement introduced Appendix A.3.
17

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Humanoid walk
Vanilla DrQv2
600
DrQv2 w/ LEAST
Score
400
200
0.6
0.8
1.0
1.2
Step
1e7
Humanoid walk
0.30
Vanilla DrQv2
DrQv2 w/ LEAST
0.25
FAU
0.20
0.15
0.10
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Step
1e7
Figure Performance on long-term training task. The curve and shade denote the mean and the standard deviation over
C.2. Ablation Study of Update-to-Data
We use TD3 as the backbone and try to improve UTD to enable fast iteration of policies to better capture the advantages of LEAST. However, can seen from Figure 16 that with the increase of UTD, the efficiency of policy optimization becomes slower, which think is due the inaccuracy of the fitting of Q function by basic algorithms such TD3, which cannot adapt to high UTD (Chen et al., 2021).
18

---

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
Humanoid
6000
5000
4000
Score
3000
2000
UTD=1 =1 1
1000
UTD -5
UTD=10 10
1
2
3
4
5
Step
1e6
Figure 16. Sensitivity to UTD. Average of 5 runs on Ant.
19


Reviews:
[{"review_id": "zt2g7GyepZ", "reviewer": "Reviewer_jmK5", "summary": "This paper introduces \"Learn to Stop\" (LEAST) to address the \"sunk cost fallacy\" in deep reinforcement learning (RL). The sunk cost fallacy refers to how RL agents must complete episodes even if the trajectory collected thus far is already poor, which ultimately provides low-quality data to the agent. LEAST allows the agent to terminate episodes early by dynamically evaluating the quality and learning potential of current trajectories based on Q-values and critic gradient information. Empirically, LEAST consistently improves sample efficiency and final performance compared to baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions and useful feedback of our work. Please find our responses to each of the concerns below.\n\n**Q1: Can you demonstrate experimentally that LEAST truncates low-quality trajectories?**\n\nWe conduct additional experiments on Ant to analyze trajectory length and quality, and will include it in the revision. Experiments below confirm that LEAST effectively truncates low-quality trajectories.\n\n**LEAST effectively truncates trajectories:** We compare the distribution of trajectory lengths stored in the replay buffer between LEAST and vanilla agents (built upon TD3) during different stages of training. Table 1a shows that LEAST stores significantly more short trajectories than the vanilla agent, especially during the early training stages.\n\nTable 1a. Percentage distribution of lengths in the buffer at different training stages (25%, 50%, 75%, and 100%). Trajectory length (Short (S): $l \\leq 333$, Medium (M): $333<l \\leq 666$, Long (L): $l > 666$):\n\n| Method | 25% Progress | 50% Progress | 75% Progress | 100% Progress |\n| --- | --- | --- | --- | --- |\n| **LEAST-TD3** | S: 27.25, M: 46.14, L: 24.61 | S: 45.85, M: 38.62, L: 15.53 | S: 23.75, M: 39.17, L: 37.11 | S: 4.42, M: 6.35, L: 89.23 |\n| Vanilla-TD3 | S: 21.17, M: 52.38, L: 26.45 | S: 8.69, M: 15.03, L: 76.28 | S: 4.47, M: 8.24, L: 87.29 | S: 1.18, M: 7.51, L: 91.31 |\n\n**Truncated trajectories are low-quality.**  We design a comparison experiment that evaluates the quality of trajectories identified for truncation by LEAST. We periodically (every 10k steps) stop the truncation to allow these \"would-be truncated\" trajectories to execute fully until their natural termination. The final cumulative rewards of these trajectories are then compared to the average rewards of trajectories under normal LEAST training. Results below show that \"would-be truncated\" trajectories achieve lower rewards across all training phases, confirming that LEAST effectively truncates suboptimal trajectories. This is further supported by suboptimal path truncation visualizations in MAZE ([ANONYMOUS LINK](https://anonymous.4open.science/r/lzvk-EF05/R_1.png)).\n\nTable 1b. Effectiveness of LEAST in truncating low-quality trajs:\n\n| Method | 25% Progress | 50% Progress | 75% Progress | 100% Progress |\n| --- | --- | --- | --- | --- |\n| **would-be-truncated** | $1619.26\\pm1033.58$ | $3361.37\\pm653.27$ | $4783.07\\pm468.63$ | $4973.48\\pm502.86$ |\n| **Baseline** | $3975.26\\pm507.91$ | $4873\\pm282.35$ | $5523\\pm361.24$ | $6376\\pm417.78$ |\n\n**Q2: The plasticity loss experiments would be more appropriately placed in an appendix...**\n\nWe will move this to the appendix and use the space to expand the analyses in appendix.\n\nThe plasticity loss experiments were included to investigate a secondary benefit of LEAST in mitigating plasticity loss. Premature convergence caused by low-quality data (primacy bias [1]) is correlated with plasticity loss [2]. Since LEAST improves buffer data quality by truncating suboptimal trajectories (Fig. 3), we hypothesized it could mitigate plasticity loss, as analyzed in Fig. 11 to provide preliminary evidence for future work.\n\n[1] The primacy bias in deep reinforcement learning. ICML'22.\n\n[2] Loss of Plasticity in Continual Deep Learning. Nature'24.\n\n**W1: The additional mechanisms appear to function more as heuristic patches...**\n\nThese auxiliary mechanisms ensure LEAST's robustness in complex environments.\n\nWhile the basic Q-based module works well on simple tasks like Hopper, but becomes less reliable in more complex environments, e.g. Ant. We propose two complementary modules: a dynamic buffer to improve threshold calculations by filtering outliers, and noise adjustments to maintain exploration despite early stopping.\n\n**W2: The paper would benefit from exploring potential tradeoffs of early stopping and exploration...**\n\nTo achieve that tradeoff, we introduce a simple yet effective noise adjustment module (which reduces the frequency of early stopping), helping to maintain exploration diversity and mitigate the risk of converging to overly short-term strategies. We hope our work can inspire research for further improvement.\n\n**W3: Clarify what \"Loss\" refers to specifically in Fig 3?**\n\nLoss refers to critic loss. We classify samples as High/Low Q based on their Q-values compared to the median Q-value in the Q buffer (and similarly for High/Low loss).\n\n**W4: Fig 5 only presents results for Ant and HC.**\n\nWe chose these tasks as they represent different complexity levels - Ant: complex; HC: simpler - allowing us to evaluate LEAST across varying challenges. We will include full results in the revision.\n\n**W5: Typos, fragments & box plot.**\n\nWe improved them in the revision. The edge of box plots indicates upper and lower bounds.\n\n---\nThanks again for reviewing our work! We hope our responses address the concerns and would appreciate your consideration in the evaluation. We are happy to provide further clarification if needed.", "claims_and_evidence": "**Claim 1: The traditional RL framework suffers from the sunk cost fallacy, leading to inefficient sampling and suboptimal policy learning.**       \n\n  * **Unsupported.** The core experiments do not show that LEAST is truncating suboptimal trajectories but instead focus on data efficiency improvements. The Maze examples in Figure 3 shows that LEAST agents have a higher fraction of high-return trajectories, but again does not show that LEAST is truncating suboptimal trajectories. \n\n**Claim 2: LEAST improves sample efficiency and final performance across different algorithms**\n  * **Supported.** Figures 6 and 9 show that algorithms enhanced with LEAST require fewer steps to reach the same performance level as their vanilla counterparts.", "methods_and_evaluation_criteria": "1. The evaluation criteria and benchmarks (MuJoCo and DMC) are standard and appropriate for the problem. The authors also make a good case for why these environments are suitable for testing their method, particularly in environments with potential dead-ends or suboptimal trajectories.\n\n1. What do the edges of the box in the box plots represent?", "theoretical_claims": "None.", "experimental_designs_or_analyses": "1. **Figure 3 is difficult to interpret.** Could you clarify what \"Loss\" refers to specifically? Is it the critic's loss, TD error, or the policy's return? Additionally, what criteria determine whether a sample is classified as \"High/Low Loss\" versus \"High/Low Q\"?\n1. Why does Figure 5 only present results for Ant and HalfCheetah? Are the results for other environments less compelling? If so, what might explain this discrepancy?\n1.  **The paper lacks empirical evidence demonstrating that LEAST actually terminates low-return trajectories early.** This verification is essential since early termination of unpromising trajectories is the fundamental motivation behind the algorithm.\n1. **The plasticity experiments seem disconnected from the main contribution.** The paper's shift to studying network plasticity appears abrupt and inadequately motivated, and these experiments feel preliminary. While Figure 11 suggests LEAST reduces plasticity loss rates, this observation may be misleading. One of LEAST's primary motivations is that completing obviously low-quality episodes provides poor training data, potentially leading to premature convergence to suboptimal policies. Would this premature convergence potentially lead to faster loss of plasticity?", "relation_to_broader_scientific_literature": "The paper appropriately positions its contribution within the RL literature, particularly in relation to sample efficiency methods and existing approaches to early stopping. The authors discuss related work in Appendix A, covering both sample efficiency techniques in Deep RL and early stopping mechanisms.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Additional Strengths:**\n\n1. LEAST is intuitive and easy to understand.\n2. The empirical validation is comprehensive.\n\n**Additional Weaknesses:**\n\n1. While the basic Q-based threshold version of LEAST presents a sound approach, the additional mechanisms appear to function more as heuristic patches rather than integrated components of the algorithm's core design.\n1. The paper would benefit from exploring potential tradeoffs of early stopping, particularly how it might affect the discovery of valuable long-term strategies that only become apparent after extended training periods.\n1. The paper contains several typos and sentence fragments that occasionally impact readability (specific examples are provided in the next section).\n\n**Author comments on these weaknesses would be appreciated, though I emphasize that these weaknesses are not critical to my evaluation.**", "other_comments_or_suggestions": "1. \"which proves that it is trapped in sub-optimality and can’t escape.\" While I would agree that the Vanilla training curve for the Medium maze hints at suboptimal convergence, it is unclear if the Vanilla training curve for the Large maze has converged yet. \"Prove\" seems like too strong of a word here. Plotting the entropy of the policy throughout training could make the argument for suboptimal convergence stronger.\n1. \"Figure 4 shows this dual-criteria threshold (ωi × εi ) significantly outperforms purely Q-based threshold\" What does \"significantly\" mean here, precisely? Figure 4 shows that the 25th percentile dual criteria threshold agents outperform the 75th percentile Q-based threshold agents, but does this constitute a significance test? (What do the edges of the box in each box plot represent? I assumed 25/75th percentile). This statement should be rephrased to explicitly state what metrics we are comparing.\n1. \"Makes it difficult to measure central tendency.\" is a sentence fragment.\n1. \"Specifically, for TD3 and SAC (Figure 6(a,b)).\" is a sentence fragment\n1. Figure 7 should have error bars\n2. Line 327: \"mwthod\" -> method\n3. line 328: \"thebox\" -> the box\n4 \"thus avoiding the agent from training via\" -> thus preventing the agent from training on\n5. Line 385: \"sequential of skills\" it is unclear what this means\n6. Line 385: \"Compare to TD3\" -> compared \n7. \"Appendix D.2 detailed introduces\" -> extra word\n1. \"We analyze the impact of the startup time of LEAST in Appendix D.4, it is a good choice to start LEAST from 10% − 20% of the total training time for MuJoCo.\" is a run-on sentence\n1. \"for the image input task\" -> tasks\n1. Line 643: broken reference to DDPG\n1. Line 796: \"Deep RL\" -> Deep RL\n1. Line 932: \"noisy daily\"\n1. \"Divide quadrants uniformly using the mean of Q and Loss of advanced agent buffer samples.\" is a sentence fragment\n1. Line 797: Broken reference for REDQ.", "questions_for_authors": "**I currently lean to reject. While experiments demonstrate LEAST's improved data efficiency, the paper lacks convincing evidence that this improvement stems from truncating low-quality trajectories as claimed.**\n\n1. Can the authors provide additional experiments showing that LEAST is indeed truncated low-quality trajectories? For instance, one could plot the distribution of trajectory lengths thoughout training for Vanilla and LEAST agents (in some fair manner). \n\n2. Could the authors contextualize the plasticity loss experiments more? My impression is that the plasticity loss experiments would be more appropriately placed in an appendix, as their relevance to the core contributions seems limited. This would free up space for more informative ablation studies that could better illuminate the algorithm's effectiveness. \n\n**If the authors address both of these points, I will consider raising my score.**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E9PJ1zsRGK", "reviewer": "Reviewer_zKrJ", "summary": "The paper proposes a novel technique for improving RL training that allows an agent to terminate an episode early if the expected return drops below a heuristic threshold. The paper aims to show that this can speed up training as it avoids filling the replay buffer with uninformative trajectories. The authors provide evidence for their claim across a variety of benchmarks and baseline algorithms.\n\n\n### Update after rebuttal period\n\nWith additional context, I support the acceptance of this paper. All major questions were adequately addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions and useful feedback of our work. Please find our responses to each of the concerns below.\n\n## Question\n\n**Q1.1: Please verify the findings with more up-to-date algorithms such as SR-SAC, CrossQ,...,e.g., repeating the experiments on Mujoco tasks with a model-free algorithm.**\n\nThanks for your suggestion. We have incorporated CrossQ as an additional backbone algorithm and conducted experiments on Hopper & HalfCheetah (with full results to be included in the revision due to time constraints). \n\nThe results show that CrossQ-LEAST consistently outperforms vanilla CrossQ in both performance & learning efficiency. This improvement may stem from CrossQ's more stable Q-value predictions compared to SAC, which helps calculate more reliable stop thresholds.\n\nTable: Score$\\uparrow$ (Convergence steps (M)$\\downarrow$ )  (Average of 3 runs):\n\n| Method | Hopper| HalfCheetah |\n| --- | --- | --- |\n| **CrossQ-LEAST** | $2853\\pm162.37$ ($3.25$) | $13607.57\\pm 115.48$ ($2.26$) |\n| CrossQ | $1839.36\\pm297.24$ ($4.71$) | $13035.81\\pm 137.16$ ($2.67$) |\n\n**Q1.2: Why wasn't LEAST evaluated with TACO or A-LIX as well? The improvements should be orthogonal?**\n\nSince both methods are based on DrQv2, we used it as our primary backbone to evaluate LEAST's general improvements. We also tested LEAST with A-LIX and found consistent performance gains across image-based domains (results to be included in revision). Besides, MBRL indeed remains an interesting direction for future work.\n\n| Method | Finger Turn Hard | Quadruped Run |\n| --- | --- | --- |\n| **A-LIX w/ LEAST** | $537\\pm19.24$ | $858.73\\pm 16.81$ |\n| A-LIX | $451\\pm32.61$  | $772.53\\pm 25.97$ |\n\n**Q2: Relevant prior work from the field of robotics seems not to be discussed.**\n\nWe appreciate this suggestion and will enhance the related work section in the revision to thoroughly discuss relevant research from the robotics field.\n\n## Weakness\n\n**W1.1: The algorithm is presented as the agent \"learning\" to avoid the sunk cost fallacy. But no learning is happening as far as I can tell.**\n\nWe appreciate your suggestion and will revise the phrasing to use \"decide\" instead of \"learn,\" as it more accurately reflects how LEAST functions. We originally used the learning analogy to illustrate the behavioral patterns that LEAST promotes.\n\n**W1.2: The sunk cost fallacy seems to be only somewhat applicable here...**\n\nOur goal is to make it more intuitive for readers to understand our core contribution: existing RL architectures often lack the capability to dynamically evaluate the potential utility of continuing a trajectory, leading agents to \"stubbornly\" execute the full trajectory regardless of its relevance or productivity within the given interaction budget. This inefficiency bears an analogy to the sunk cost fallacy, where agents implicitly persist in a trajectory simply because it has already been initiated, without reassessing whether it remains promising or worthwhile. While the sunk cost fallacy is traditionally associated with human decision-making, we draw this analogy to highlight how LEAST's dynamic stopping mechanism addresses a similar inefficiency in RL.\n\n**W2: In other tasks with less well-shaped rewards & early reset structure, the method might limit exploration to a problematic extent?**\n\nLEAST's adaptive stopping mechanism functions independently of environment-level reinitialization, as it can detect suboptimal states before the environment triggers a reset. In locomotion tasks, e.g., LEAST identifies patterns like in the agent instability that indicate imminent failure, proactively stopping the episode before environment-level reinitialization occurs—particularly during early training stages. This proactive stopping enables LEAST to leverage its dynamic stopping mechanism to improve learning efficiency, even in environments with early reinitialization (as demonstrated in Figs 6, 7, 9). \n\nTo address the exploration limitations, we introduce a simple yet effective method, dynamic exploration noise adjustment, and find that it effectively helps agents escape suboptimal trajectories to explore more diverse behaviors. Thus, in environments with less well-shaped rewards & early reinitialization structure (e.g., PointMaze), LEAST still demonstrates robust performance, further confirming the method’s general applicability. \n\n*We hope our work can inspire future research to build upon this foundation for more challenging scenarios. Exploring further improvements to LEAST's exploration capabilities remains an exciting direction for future work.*\n\n---\n\nWe would like to thank the reviewer once again for the time and effort in reviewing our work! We are happy to provide further clarification if you have any additional questions. We hope that our responses adequately address your concerns, and we would greatly appreciate it if you could kindly consider reflecting on this in the evaluation.", "claims_and_evidence": "The paper is empirical in nature and provides a variety of evidence for its claims. The evidence seems sufficient for the claims.", "methods_and_evaluation_criteria": "The bench-marking covers several different RL problems and algorithms. A small (rather nitpicky) caveat is that the baseline RL algorithms use are relatively old and have known weaknesses such as problems with learning stability. All considered algorithms are 4 years old at this point. To address this, I would encourage the authors to verify their findings with more up-to-date architectures and algorithms such as SR-SAC [1], BRO [2], Simba [3], CrossQ [4], TD-MPC2 [5], or MAD-TD [6]. Obviously I am not asking the authors to provide evidence on all of these, but for example repeating the experiments on Mujoco tasks with a model-free algorithm shouldn't be too much of an ask. Model-based algorithms such as TD-MPC2, Dreamer or MAD-TD might be another interesting direction, as the gathered data influences both model and value function learning. In the image based domains, why wasn't LEAST evaluated with TACO or A-LIX as well? The improvements should be orthogonal?\n\nOne additional question for clarification: in the reported scores, were the algorithms evaluated with or without early resets? I think it would be important to verify that the additional performance is not due to the algorithms being able to avoid \"difficult\" parts of the state space they might naturally have to end up in. Clarifying this is a requirement for me supporting the acceptance of the paper!\n\n[1] Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier, D'Oro et al, ICLR 2023 https://openreview.net/forum?id=OpC-9aBBVJe\n\n[2] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, Nauman et al, NeurIPS 2024, https://openreview.net/forum?id=fu0xdh4aEJ&noteId=It2lExTd92\n\n[3] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning, Lee et al, ICLR 2025, https://openreview.net/forum?id=jXLiDKsuDo\n\n[4] CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity, Bhatt et al, ICLR 2024, https://openreview.net/forum?id=PczQtTsTIX\n\n[5] TD-MPC2: Scalable, Robust World Models for Continuous Control, Hansen et al, ICLR 2024, https://openreview.net/forum?id=Oxh5CstDJU\n\n[6] MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL, Voelcker et al., ICLR 2025, https://openreview.net/forum?id=6RtRsg8ZV1", "theoretical_claims": "No theoretical claims were made.", "experimental_designs_or_analyses": "Aside from the concerns raised above, the experimental design is well done.", "relation_to_broader_scientific_literature": "They seem to fit the literature.", "essential_references_not_discussed": "Relevant prior work from the field of robotics seems to not be discussed, such as [1] and follow up work. As robotics is a field where limiting long episodes has strong practical relevance, this seems important.\n\n[1] Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning, Eysenbach et al, ICLR 2018 https://openreview.net/forum?id=S1vuO-bCW", "other_strengths_and_weaknesses": "I have two problems with the presentation of the paper, but these are relatively minor. First of all, the algorithm is presented as the agent \"learning\" to avoid the sunk cost fallacy. However, LEAST is a fixed heuristic, so no learning is happening as far as I can tell. In addition, the sunk cost fallacy seems to be only somewhat applicable here, as agents do not normally have a choice but to continue in a trajectory until the end.\n\nThe other problem is that the authors mostly consider environments which already include early resets, such as the mujoco tasks. In these, avoiding \"bad\" areas of the state space is indeed optimal, as the agent is most likely facing an early reset anyways. However, in other tasks with less well shaped rewards and early reset structure, the method might limit exploration to a problematic extend. I would ask the authors to comment on this more clearly. The authors already comment on some exploration related problems in 3.2, but I believe that the chosen test tasks might lead to optimistic conclusions on the methods peformance.", "other_comments_or_suggestions": "none", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJD7jb7z1R", "reviewer": "Reviewer_Rcz7", "summary": "The paper introduces the learn to stop (LEAST) heuristic for online off-policy reinforcement learning (RL). \nThe core idea is the proposition of an adaptive stopping mechanism to prohibit including unhelpful-to-the-learning-task transitions to the replay buffer. The heuristics is then experimentally evaluated on a variety of aspects with four MuJoCo tasks and DMC tasks.\n\n## update after rebuttal\nScore increased. See rebuttal comment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful suggestions and constructive feedback on our work. Please find our responses to each of the concerns below.\n\n**Q1: What do you mean in Background by \"In this paper, we investigate whether deep RL agents deployed on cumulative reward maximization tasks also suffer from sunk cost fallacy.\"?**\n\nWe investigate whether RL agents exhibit behavior analogous to the sunk cost fallacy in reward maximization tasks, where agents inefficiently persist in executing trajectories despite diminishing returns will cause more losses, similar to how investors might hold onto losing investments. Through this analysis, we aim to identify and address fundamental inefficiencies in conventional RL architectures.\n\n**Q2: What makes the introduced heuristic special for deep RL in comparison to its importance for \"non-deep\" RL?**\n\nWe would like to clarify that our proposed methodology is general for RL (not specific to Deep RL).\n\nWe validate our method's generality through experiments on tabular Q-learning in two MiniGrid navigation tasks [1] with default parameters. Results below measure the episode number required to converge to the optimal policy. LEAST significantly improves learning efficiency compared to standard Q-learning (**5%-14%** reduction in episodes required to converge), demonstrating its effectiveness in improving learning efficiency in simpler, discrete non-deep RL settings.\n\n| Method | 6x6 with obstables ($\\downarrow$) | 8x8 with obstables ($\\downarrow$) |\n| --- | --- | --- |\n| **Q learning w/ LEAST** | $428.0\\pm21.3$ | $495.3\\pm24.7$ |\n| Q learning | $494.7\\pm28.9$ | $523.3\\pm26.3$ |\n\n**Why we mainly discuss deep RL:** While non-deep RL provides a valuable testbed for validating the algorithms clearly,  our main focus is to make deep RL algorithms learn better and use data more efficiently, since Deep RL is critical for solving complex, high-dimensional problems and has unique challenges and practical relevance in real-world applications. We aim to validate our approach in these demanding scenarios while encouraging future theoretical analysis in simpler RL paradigms.\n\nWe hope our work can inspire exploration in non-deep RL, e.g. tabular & linear methods for theoretical analysis.\n\n[1] Open source GitHub repo: Minigrid-RL (default setting)\n\n**Q3: What do you mean by \"In this section, we conduct comprehensive experiments to evaluate whether LEAST module is necessary for Deep RL.\"?**\n\nWe will revise the sentence for clarity in the revised version. To clarify, in this section, we aim to evaluate the necessity of LEAST by investigating whether it provides significant improvements in the performance & efficiency of baselines.\n\n**Q4: ... why the proposed criteria of the LEAST are a good choice in general? ... Especially why is this better than a possible multi-criteria.**\n\nThe proposed dual-criteria based on Q-values & loss offer the following major advantages, making it a robust, versatile, and general choice for our LEAST framework:\n\n- **Robustness**. Q-values assess trajectory quality through expected rewards, while critic loss indicates the agent's understanding (familiarity) of transitions. These complementary metrics provide comprehensive evaluation criteria for stopping decisions.\n- **Versatility**. Q-values & loss are core metrics in RL algorithms, making them algorithm-agnostic & environment-independent, unlike entropy (SAC-specific) or scene-specific metrics. This enables the dual-criteria to work across different tasks and algorithms with minimal changes.\n- **Simplicity**. Q-values & loss are readily computed metrics that require no extra computation (e.g., network modules), making the criteria efficient to implement.\n\nWe explored various multi-criteria but found none that consistently outperformed the dual-criteria in efficiency and task independence. Our motivation is to provide a fundamental framework that is simple, effective, and broadly applicable across diverse tasks and algorithms, and to encourage future research into more sophisticated multi-criteria approaches.\n\n**W1: The presented background has nothing to do with \"Deep\" RL, it describes RL basics. It seems reasonable...**\n\nWe revised this paragraph in the background section to better reflect the context of deep RL.\n\n**C1: The connection between the observed problem & the sunk-cost fallacy seems far-fetched...**\n\nOur analogy here focuses on the behavioral similarity: the implicit commitment to completing trajectories without evaluating their future benefit, although the classical sunk-cost fallacy typically involves emotional attachment. The absence of early stopping mechanisms in existing RL architectures leads to decision-making inertia, resembling the sunk-cost fallacy in its effect on efficiency.\n\n---\nThank you again for reviewing our work. We will refine the typos and hope our responses address the concerns and would appreciate your consideration in the evaluation. We are happy to provide further clarification if needed.", "claims_and_evidence": "The paper states its main contributions as \n(1) identifying a critical limitation of current RL frameworks\n(2) introduction of a stopping heuristic with general benefits\n(3) experimental validation of the proposed heuristic.\n\nClaim (1) seems to be supported by clear and convincing evidence.\nClaim (3) is supported by the experiments presented in the paper.\n\nClaim (2) is not entirely clear to me. See also Question for Authors.", "methods_and_evaluation_criteria": "Looks appropriate except open question in Question for Authors.", "theoretical_claims": "Theoretical foundation for the presented heuristic would be nice, but is not included in the paper.", "experimental_designs_or_analyses": "The experimental verification looks valid for the presented results.", "relation_to_broader_scientific_literature": "The paper contributes to the online RL literature. Due to its nature to terminate episodes early, potentially saving computation costs, it might be relevant to computationally heavy experiments.", "essential_references_not_discussed": "Not relevant.", "other_strengths_and_weaknesses": "The presented background (often called preliminaries) has nothing to do with \"Deep\" RL, it describes RL basics. It seems reasonable though that the addressed problem is especially relevant for Deep RL.\n\n\nAppendix B.4 is an exact repetition of a paragraph in Section 2.", "other_comments_or_suggestions": "To me, the connection between the observed problem and the sunk-cost fallacy seems far-fetched. Sunk-cost fallacy depends on the emotional attachment to whatever resource is already sunk. The continuation of an episode until termination stems rather from a missing mechanism instead of an attached to the current episode.\n\nThe referencing of subsection with \"§\" does not fit ICML standards.\n\nFigure 5 and 13 have surprising format sizes. \n\nTypo: mwthod (327).", "questions_for_authors": "What do you mean by \"In this paper, we investigate whether deep RL agents deployed on cumulative reward maximization tasks also suffer from sunk cost fallacy.\"?\n\nWhat exactly makes the introduced heuristic special for deep RL in comparison to its importance for \"non-deep\" RL?\n\nWhat do you mean by \"In this section, we conduct comprehensive experiments to evaluate whether LEAST module is necessary for Deep RL.\"? Introducing a trainings heuristic can be helpful, but I do not understand what you mean by necessary. Please elaborate.\n\nCan you summarize comprehensibly why the proposed criteria of the LEAST heuristic is a good choice in general for stopping episodes? The experimental evidence seems not entirely clear. Especially why the proposed dual-criteria would be better than possible multi-criteria.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gslXJYDS67", "reviewer": "Reviewer_4RG4", "summary": "The paper proposes a simple technique to avoid the issue of deep RL agents collecting familiar low-rewarding data to avoid the issue of re-sampling this data during training as a way to improve sample efficiency of deep RL agents. The paper integrates the algorithm in a variety of algorithms, applies it on different benchmarks including image data, and conducts an ablation.\n\n## update after rebuttal\n\nI am leaning on acceptance since my questions were adequately addressed and the paper presents an interesting idea.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful suggestions and helpful feedback of our work, and for the recognition of our work's motivation, performance, and potential academic impact.\n\n**Q1:  (1) Given that the training loss is so critical to filter data, what do the authors think about the unreliability of the loss function in off-policy algorithms and (2) how that may be affecting LEAST ?**\n\n(1) To mitigate the unreliability of loss calculations, LEAST mitigates this issue by using the median of the loss values rather than relying on a single loss value per transaction, which reduces sensitivity to noise & biases. It will also be an interesting future direction to consider more robust loss calculation. \n\n(2) While LEAST shows strong performance in the current experimental settings, an interesting direction for future work would be to investigate its behavior in tasks requiring even longer-term learning. In such scenarios, the potential instability of loss calculations might affect the episode-stopping threshold, which could slightly reduce the method’s efficiency in certain edge cases. It is an interesting future direction for further improving the robustness of loss calculations, which can further extend LEAST's applicability and improve its stability across a wider range of tasks.\n\n**Q2: How do you think performance would change if instead of storing the Q values in B_Q, the MC returns were saved?**\n\nThank you for the suggestion! MC returns could be a potential metric. However, it can introduce two main challenges for implementing this within LEAST’s framework:\n\n- *Interaction efficiency:* MC return requires repeatedly resetting to specific states to compute estimates from multiple rollouts to reduce variance in its estimation, which can be sample-inefficient.\n- *Suffer from biased estimation:* MC return heavily depends on observing complete trajectories up to their true terminal states. When trajectories are truncated prematurely (as in LEAST), the missing components of the return are treated as zero, leading to biased and inaccurate calculations.\n\nWe hope future work will explore more advanced estimations based on MC returns to effectively address the aforementioned challenges.\n\n**Q3: Isn’t the issue of unchanging policy true for even non-LEAST methods? It's a little unclear to me why LEAST has this issue. Given this seems like a universal issue...**\n\nWhile this problem is indeed common in RL, it is particularly critical for LEAST due to its unique mechanism.\n\nThe motivation for proposing a noise adjustment schedule stems from specific observations during our experiments. Although the Q- & loss-based stopping threshold provides core benefits, LEAST occasionally triggers frequent resets during early training stages, which can hinder long-term exploration (crucial for extended execution tasks like humanoid walk).\n\nThus, we introduced a simple yet effective noise mechanism controlled by premature reset frequency, aiming to stabilize exploration, especially during early training. When integrated with TD3, the results showed minimal improvement, suggesting that noise adjustment may be less beneficial for vanilla deep RL compared to its impact on LEAST.\n\nTable. Performance $\\uparrow$ (Average of 3 runs):\n\n| Method | Ant | Humanoid |\n| --- | --- | --- |\n| **TD3 w/ noise adjustment** | $5704 \\pm 145.23$ | $4591.28\\pm 224.82$ |\n| TD3 | $5682.13\\pm 129.48$ | $4616.54\\pm 273.69$ |\n\n**Q4: Nothing to really comment but something of potential interest to authors ...**\n\nThis is very interesting! We will analyze it in the related work of the new version.\n\n**W1: Fig 7 does not include information about std, variance etc... even though LEAST clearly has better sample efficiency.**\n\nThanks for your suggestion. We used 5 random seeds, and we will update the caption and add error bars in Fig 7. \n\n**W2: The discussion on entropy aware dynamic buffer size & adjusting exploration noise seems quite ad-hoc... should have some discussion**\n\nTo ensure robust episode truncation in complex environments, we implement two key mechanisms beyond the core Q-value & loss thresholds: (1) a dynamic buffer that filters outliers to improve threshold stability & accuracy when Q-values become unreliable, and (2) to prevent exploration limitations from frequent resets, we introduce an action noise mechanism that adaptively balances exploration and stopping behavior.\n\nWe will elaborate more in the revision.\n\n**W3: … While LEAST does help plasticity it is a little unclear whether this is because it fits higher quality data or .... this should be specified.**\n\nWe would like to clarify that LEAST improves plasticity not by reducing the number of updates, instead, it helps the agent to focus on more informative samples which accelerates adaptation and helps it explore better policies.\n\n---\nWe would like to thank the reviewer once again for the time and effort in reviewing our work! We are happy to provide further clarification if you have any additional questions.", "claims_and_evidence": "Yes, in general, paper's claims are accompanied with convincing evidence.\nFew issues:\n- The graph on Figure 7 does not include any information about standard deviation, variance etc. Nor does it mention the number of trials, which I am assuming 5 * # of environments. This information is critical even though LEAST clearly has better sample efficiency.\n- The discussion on entropy aware dynamic buffer size and adjusting exploration noise (Section 3) seems quite ad-hoc. The previous paragraphs are well-motivated and those are well-analyzed in the empirical section, but these other sections seem out-of-place. I see there is some evaluation in the Appendix. I think the main paper should have some discussion about this.\n- Section 4.3 mentions that LEAST helps “avoid the agent from training via useless transitions too often”. While LEAST does help plasticity it is a little unclear to me whether this is because it fits higher quality data or because there are fewer data samples since LEAST filters them out. It seems like an algorithm’s plasticity could increase if it just updated less often. I presume the number of gradient steps are the same, which means it would be the former. If this is the case, this should be specified.", "methods_and_evaluation_criteria": "Yes, the paper integrates the proposed algorithm in different settings, baseline algorithms (TD3, SAC, DrQv2), and in visual/non-visual environments that are commonly used in the deep RL literature.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the paper conducts a reasonable analysis except for one point mentioned above (copy-pasting here): The graph on Figure 7 does not include any information about standard deviation, variance etc. Nor does it mention the number of trials, which I am assuming 5 * # of environments. This information is critical even though LEAST clearly has better sample efficiency.", "relation_to_broader_scientific_literature": "The paper falls under the broad category of improving the sample efficiency of RL algorithms in robotics simulation tasks. It revisits how deep RL algorithms are fundamentally designed to use the data they collect and suggests a data filtration process to use higher quality data during training.", "essential_references_not_discussed": "It would be useful to make some references to algorithms that sample data differently instead of filtering data. For example, prioritized experience replay (Schaul 2015) or event tables (Kompella 2023). These methods seems similar in spirit of updating the agent using better data.", "other_strengths_and_weaknesses": "Strengths:\n- The algorithm is simple to implement and incorporate into any RL algorithm\n- It is evaluated on visual and non-visual environments with different baselines.\n- It conducts a thorough empirical analysis and ablation\n\nWeaknesses:\n- See above\n- The last two lines in the first paragraph Section 3.2 are very unclear and seem to just be filled with jargon. It would be better to simplify.\n- The paragraphs on dynamic buffer size and exploration noise are quite confusing. The notation used also is slightly abused which makes things tricky to parse. Are the last h episodes removed? If not, which ones? What does UTD mean? It should be specified.\n- First few lines on page 6 discusses why LEAST eventually does better in HalfCheetah, but it does not discuss why it may be worse during early training. I think that is the real anomaly which should be explained.", "other_comments_or_suggestions": "Page 6 paragraphs have some typos and the dynamic buffer size/exploration noise paragraphs are unclear.", "questions_for_authors": "- Given that the training loss is so critical to filter data, what do the authors think about the unreliability of the loss function in off-policy algorithms and how that may be affecting LEAST [1]?\n- How do you think performance would change if instead of storing the Q values in B_Q, the Monte Carlo returns were saved?\n- Isn’t the issue of unchanging policy true for even non-LEAST methods? It's a little unclear to me why LEAST has this issue. Given this seems like a universal issue, it seems like LEAST may be better since it has an improved exploration strategy.\n- Nothing to really comment but something of potential interest to authors is [2], which is for the average reward setting where the agent tries to figure out when it should reset the episode, where there is a cost associated with resetting the episode.\n\n\n[1] Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error. Fujimoto et al. 2022.\n\n[2] RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning. Hisaki et al. 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces LEAST, a mechanism designed to improve sample efficiency in deep reinforcement learning by allowing agents to terminate unproductive episodes early. The core idea is framed as overcoming the "sunk cost fallacy," where agents continue episodes even when they are unlikely to yield useful data, thus polluting the replay buffer. LEAST uses Q-values and critic loss statistics to decide when to stop an episode prematurely.   

Initial reviews were mixed but generally acknowledged the potential value of the proposed method. Reviewers appreciated the simplicity and intuition behind LEAST, along with its empirical validation across various benchmarks and algorithms. However, several concerns were raised. Reviewer jmK5 questioned whether the method demonstrably truncates low-quality trajectories as claimed and found the experiments on network plasticity somewhat disconnected from the main contribution. Reviewer zKrJ, while positive, pointed out the use of older baseline algorithms and questioned the aptness of the "sunk cost fallacy" terminology, also requesting clarification on the evaluation protocol regarding early resets. Reviewer Rcz7 also found the sunk cost analogy far-fetched and questioned the general applicability and necessity of the specific heuristic criteria used by LEAST. Reviewer 4RG4 raised concerns about the potential unreliability of using training loss as a criterion and the somewhat ad-hoc justification for components like the dynamic buffer and exploration noise adjustment.   

The authors provided a detailed rebuttal, addressing the key concerns effectively. They presented additional experiments specifically demonstrating that LEAST does indeed truncate shorter, lower-quality trajectories, satisfying Reviewer jmK5's main reservation. They agreed to move the plasticity experiments to the appendix. In response to Reviewer zKrJ, they showed positive results incorporating LEAST with newer algorithms like CrossQ and A-LIX and crucially clarified that evaluations were performed without allowing LEAST agents to benefit from early resets during the evaluation phase, ensuring a fair comparison. They defended the choice of dual-criteria (Q-value and loss) for its generality and simplicity, while also demonstrating LEAST's applicability in non-deep RL settings for Reviewer Rcz7. While acknowledging the "sunk cost fallacy" analogy might not be a perfect fit (lacking the emotional component), they argued for its intuitive value in conveying the problem of inflexible trajectory execution and agreed to refine the explanation. Concerns from Reviewer 4RG4 regarding loss unreliability and the motivation for auxiliary mechanisms were also addressed with further explanations.   

Following the rebuttal, all reviewers acknowledged the authors' responses. Reviewer jmK5 and Reviewer Rcz7 explicitly raised their scores to Weak Accept. Reviewer zKrJ and Reviewer 4RG4 maintained their Accept recommendations, expressing satisfaction with the clarifications. The consensus among reviewers shifted positively, recognizing the paper's contribution in addressing an inefficiency in DRL sample collection with a novel and practical approach, supported by strong empirical evidence after the rebuttal.
Decision: Accept (poster)