Repository: nextflow-io/nextflow
Pull request #5868: Fix task progress percentage with failed tasks
Author: login: bentsherman
Created: 2025-03-10T17:06:35Z | Merged: 2025-05-08T14:21:17Z

Spun off from #4588 to provide a more focused change

Currently, when a task fails, it is included in the "complete" and "total" task count per process. As a result, the percentage actually increases when tasks fail, even though no progress has been made. It also distorts the total number of "real tasks" vs "task attempts" in the final log.

This PR removes the "failed" count from the complete/total task counts, and adds the "ignored" count in its place. The rationale is that every failed task will either be retried or ignored, or the run will terminate and it won't matter. Of these failures, only the ignored tasks should count towards progress. Retries that succeed will be marked as succeeded and count towards progress, otherwise they shouldn't.

To see the difference, run this pipeline with and without these changes:
```groovy
process foo {
  errorStrategy { sleep(5000) ; task.exitStatus==1 && task.attempt==1 ? 'retry' : 'ignore' }

  input:
  val x

  script:
  """
  sleep ${x * 3}
  exit 1
  """
}

workflow {
  Channel.of(1, 2, 3) | foo
}
```

This pipeline executes three tasks that fail, are retried, fail again, and ignored, with some delays to help you see the progress.

Without this PR, the process will appear to make progress as tasks fail and are retried. The final task count will be 6, even though there were only 3 "real tasks" to complete.

With this PR, the progress will remain at 0 until the tasks are ignored towards the end. The final task count will be 3, with the retried and ignored count on the side.

@jorgee if you agree with these changes, please approve so that I can merge. Let me know if you have any questions


Diff:
diff --git a/modules/nextflow/src/main/groovy/nextflow/trace/AnsiLogObserver.groovy b/modules/nextflow/src/main/groovy/nextflow/trace/AnsiLogObserver.groovy
index ff05d8f949..fc1a40a8d8 100644
--- a/modules/nextflow/src/main/groovy/nextflow/trace/AnsiLogObserver.groovy
+++ b/modules/nextflow/src/main/groovy/nextflow/trace/AnsiLogObserver.groovy
@@ -475,8 +475,8 @@ class AnsiLogObserver implements TraceObserver {
             term.a(Attribute.INTENSITY_FAINT).a(", cached: $stats.cached").reset()
         if( stats.stored )
             term.a(", stored: $stats.stored")
-        if( stats.failed )
-            term.a(", failed: $stats.failed")
+        if( stats.ignored )
+            term.a(", ignored: $stats.ignored")
         if( stats.retries )
             term.a(", retries: $stats.retries")
         // Show red cross ('✘') or green tick ('✔') according to status
diff --git a/modules/nextflow/src/main/groovy/nextflow/trace/ProgressRecord.groovy b/modules/nextflow/src/main/groovy/nextflow/trace/ProgressRecord.groovy
index 8c73ce2318..a0f1e71caa 100644
--- a/modules/nextflow/src/main/groovy/nextflow/trace/ProgressRecord.groovy
+++ b/modules/nextflow/src/main/groovy/nextflow/trace/ProgressRecord.groovy
@@ -38,7 +38,7 @@ class ProgressRecord implements Cloneable {
     int running     // number of tasks whose execution started
     int succeeded   // number of tasks whose execution completed successfully
     int cached      // number of tasks whose execution
-    int failed
+    int failed      // number of failed tasks -- includes ignored and retried
     int aborted
     int stored
     int ignored
@@ -61,13 +61,15 @@ class ProgressRecord implements Cloneable {
         this.taskName = processName
     }
 
+    // failed tasks that were not retried are included in the total count
     int getTotalCount() {
-        pending+ submitted+ running+
-           succeeded+ failed+ cached+ stored + aborted
+        pending + submitted + running +
+           succeeded + failed - retries + cached + stored + aborted
     }
 
+    // only failed tasks that were ignored are included in the completed count
     int getCompletedCount() {
-        succeeded+ failed+ cached+ stored
+        succeeded + ignored + cached + stored
     }
 
     @Override
diff --git a/modules/nextflow/src/main/groovy/nextflow/trace/WorkflowStats.groovy b/modules/nextflow/src/main/groovy/nextflow/trace/WorkflowStats.groovy
index 6fe97c2431..9cd5d37bec 100644
--- a/modules/nextflow/src/main/groovy/nextflow/trace/WorkflowStats.groovy
+++ b/modules/nextflow/src/main/groovy/nextflow/trace/WorkflowStats.groovy
@@ -97,6 +97,10 @@ class WorkflowStats implements Cloneable {
         x >= 0 ? x : 0
     }
 
+    int getTotalCount() {
+        gtz(succeededCount + cachedCount + ignoredCount + effectiveFailedCount + abortedCount)
+    }
+
     String getSucceedCountFmt() {
         INTEGER_FMT.format(gtz(succeededCount))
     }
@@ -114,23 +118,23 @@ class WorkflowStats implements Cloneable {
     }
 
     float getSucceedPct() {
-        int tot = gtz(succeededCount + cachedCount + ignoredCount + failedCount)
+        int tot = getTotalCount()
         tot ? Math.round(succeededCount / tot * 10000.0 as float) / 100.0 as float : 0
     }
 
     float getCachedPct() {
-        def tot = gtz(succeededCount + cachedCount + ignoredCount + failedCount)
+        int tot = getTotalCount()
         tot ? Math.round(gtz(cachedCount) / tot * 10000.0 as float) / 100.0 as float : 0
     }
 
     float getIgnoredPct() {
-        def tot = gtz(succeededCount + cachedCount + ignoredCount + failedCount)
+        int tot = getTotalCount()
         tot ? Math.round(gtz(ignoredCount) / tot * 10000.0 as float) / 100.0 as float : 0
     }
 
-    float getFailedPct() {
-        def tot = gtz(succeededCount + cachedCount + ignoredCount + failedCount)
-        tot ? Math.round(gtz(failedCount) / tot * 10000.0 as float) / 100.0 as float : 0
+    float getEffectiveFailedPct() {
+        int tot = getTotalCount()
+        tot ? Math.round(gtz(effectiveFailedCount) / tot * 10000.0 as float) / 100.0 as float : 0
     }
 
     protected Duration makeDuration(long value) {
@@ -164,10 +168,15 @@ class WorkflowStats implements Cloneable {
     int getSucceededCount() { gtz(succeededCount) }
 
     /**
-     * @return Failed tasks count
+     * @return Failed tasks count (includes ignored and retried)
      */
     int getFailedCount() { gtz(failedCount) }
 
+    /**
+     * @return "Effective" failed tasks count (excludes ignored and retried)
+     */
+    int getEffectiveFailedCount() { gtz(failedCount - ignoredCount - retriesCount) }
+
     /**
      * @return Ignored tasks count
      */
diff --git a/modules/nextflow/src/main/resources/nextflow/trace/ReportTemplate.html b/modules/nextflow/src/main/resources/nextflow/trace/ReportTemplate.html
index 0921415427..d02d2638c1 100644
--- a/modules/nextflow/src/main/resources/nextflow/trace/ReportTemplate.html
+++ b/modules/nextflow/src/main/resources/nextflow/trace/ReportTemplate.html
@@ -145,8 +145,8 @@ <h4>Workflow execution completed unsuccessfully!</h4>
           <div class="progress" style="height: 1.6rem; margin: 1.2rem auto; border-radius: 0.20rem;">
             <div style="width: ${workflow.stats.succeedPct}%" class="progress-bar bg-success" data-toggle="tooltip" data-placement="top" title="$workflow.stats.succeedCount tasks succeeded"><span class="text-truncate">&nbsp; $workflow.stats.succeedCount succeeded &nbsp;</span></div>
             <div style="width: ${workflow.stats.cachedPct}%" class="progress-bar bg-secondary" data-toggle="tooltip" data-placement="top" title="$workflow.stats.cachedCount tasks were cached"><span class="text-truncate">&nbsp; $workflow.stats.cachedCount cached &nbsp;</span></div>
-            <div style="width: ${workflow.stats.ignoredPct}%" class="progress-bar bg-warning" data-toggle="tooltip" data-placement="top" title="$workflow.stats.ignoredCount tasks reported and error and were ignored"><span class="text-truncate">&nbsp; $workflow.stats.ignoredCount ignored &nbsp;</span></div>
-            <div style="width: ${workflow.stats.failedPct}%" class="progress-bar bg-danger" data-toggle="tooltip" data-placement="top" title="$workflow.stats.failedCount tasks failed"><span class="text-truncate">&nbsp; $workflow.stats.failedCount failed &nbsp;</span></div>
+            <div style="width: ${workflow.stats.ignoredPct}%" class="progress-bar bg-warning" data-toggle="tooltip" data-placement="top" title="$workflow.stats.ignoredCount tasks failed and were ignored"><span class="text-truncate">&nbsp; $workflow.stats.ignoredCount ignored &nbsp;</span></div>
+            <div style="width: ${workflow.stats.effectiveFailedPct}%" class="progress-bar bg-danger" data-toggle="tooltip" data-placement="top" title="$workflow.stats.effectiveFailedCount tasks failed"><span class="text-truncate">&nbsp; $workflow.stats.effectiveFailedCount failed ($workflow.stats.retriesCount retries) &nbsp;</span></div>
           </div>
         </dl>
 
diff --git a/modules/nextflow/src/test/groovy/nextflow/trace/ProgressRecordTest.groovy b/modules/nextflow/src/test/groovy/nextflow/trace/ProgressRecordTest.groovy
index e8fecc07b5..c3aecb26dc 100644
--- a/modules/nextflow/src/test/groovy/nextflow/trace/ProgressRecordTest.groovy
+++ b/modules/nextflow/src/test/groovy/nextflow/trace/ProgressRecordTest.groovy
@@ -53,30 +53,34 @@ class ProgressRecordTest extends Specification {
 
     def 'should get counts' () {
         given:
-        def PENDING =1
-        def SUBMITTED =2
-        def RUNNING =3
-        def SUCCEEDED =4
-        def FAILED =5
-        def CACHED =6
-        def STORED =7
-        def ABORTED = 8
+        def PENDING = 1
+        def SUBMITTED = 2
+        def RUNNING = 3
+        def SUCCEEDED = 4
+        def FAILED = 7
+        def IGNORED = 5
+        def RETRIES = 1
+        def CACHED = 7
+        def STORED = 8
+        def ABORTED = 9
         and:
         def rec = new ProgressRecord(10, 'foo')
 
         when:
-        rec.pending =PENDING
-        rec.submitted =SUBMITTED
-        rec.running =RUNNING
-        rec.succeeded =SUCCEEDED
-        rec.failed =FAILED
-        rec.cached =CACHED
-        rec.stored =STORED
+        rec.pending = PENDING
+        rec.submitted = SUBMITTED
+        rec.running = RUNNING
+        rec.succeeded = SUCCEEDED
+        rec.failed = FAILED
+        rec.ignored = IGNORED
+        rec.retries = RETRIES
+        rec.cached = CACHED
+        rec.stored = STORED
         rec.aborted = ABORTED
 
         then:
-        rec.getCompletedCount() == SUCCEEDED+ FAILED+ CACHED+ STORED
-        rec.getTotalCount() == PENDING+ SUBMITTED+ RUNNING + SUCCEEDED+ FAILED+ CACHED+ STORED+ ABORTED
+        rec.getCompletedCount() == SUCCEEDED + IGNORED + CACHED + STORED
+        rec.getTotalCount() == PENDING + SUBMITTED + RUNNING  + SUCCEEDED + FAILED - RETRIES + CACHED + STORED + ABORTED
     }
 
 
diff --git a/modules/nextflow/src/test/groovy/nextflow/trace/WorkflowStatsTest.groovy b/modules/nextflow/src/test/groovy/nextflow/trace/WorkflowStatsTest.groovy
index 6139f1b7fb..b2b74a5b81 100644
--- a/modules/nextflow/src/test/groovy/nextflow/trace/WorkflowStatsTest.groovy
+++ b/modules/nextflow/src/test/groovy/nextflow/trace/WorkflowStatsTest.groovy
@@ -166,13 +166,13 @@ class WorkflowStatsTest extends Specification {
 
     def 'should return task percents' () {
         given:
-        def stats = new WorkflowStats(succeededCount: 20, cachedCount: 40, ignoredCount: 60, failedCount: 80)
+        def stats = new WorkflowStats(succeededCount: 20, cachedCount: 40, ignoredCount: 60, failedCount: 140)
 
         expect: 
         stats.getSucceedPct() == 10.0f
         stats.getCachedPct() == 20.0f
         stats.getIgnoredPct() == 30.0f
-        stats.getFailedPct() == 40.0f
+        stats.getEffectiveFailedPct() == 40.0f
 
     }
 


Reviews:
user: login: bentsherman
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T15:13:39Z
user: login: jorgee
body: There is a test failing, adding suggestions to fix it. Please check if they are correct
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T13:47:36Z
user: login: bentsherman
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T13:13:57Z
user: login: jorgee
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T13:28:56Z

Review comments:
diff_hunk: @@ -114,22 +114,17 @@ class WorkflowStats implements Cloneable {
     }
 
     float getSucceedPct() {
-        int tot = gtz(succeededCount + cachedCount + ignoredCount + failedCount)
+        int tot = gtz(succeededCount + cachedCount + failedCount)
path: modules/nextflow/src/main/groovy/nextflow/trace/WorkflowStats.groovy
user: login: bentsherman
body: This total should include the aborted count too right? Since this total is used four times I would just factor it into a separate `getTotalCount()` function
created_at: 2025-03-13T15:13:39Z
author_association: MEMBER
diff_hunk: @@ -61,13 +61,15 @@ class ProgressRecord implements Cloneable {
         this.taskName = processName
     }
 
+    // failed tasks that were not retried are included in the total count
     int getTotalCount() {
-        pending+ submitted+ running+
-           succeeded+ failed+ cached+ stored + aborted
+        pending + submitted + running +
+           succeeded + failed - retries + cached + stored + aborted
path: modules/nextflow/src/main/groovy/nextflow/trace/ProgressRecord.groovy
user: login: jorgee
body: ```suggestion
           succeeded + ignored + failed - retries + cached + stored + aborted
```
Missing ignored
created_at: 2025-05-06T13:44:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,30 +53,32 @@ class ProgressRecordTest extends Specification {
 
     def 'should get counts' () {
         given:
-        def PENDING =1
-        def SUBMITTED =2
-        def RUNNING =3
-        def SUCCEEDED =4
-        def FAILED =5
-        def CACHED =6
-        def STORED =7
-        def ABORTED = 8
+        def PENDING = 1
+        def SUBMITTED = 2
+        def RUNNING = 3
+        def SUCCEEDED = 4
+        def FAILED = 5
+        def IGNORED = 6
+        def CACHED = 7
+        def STORED = 8
+        def ABORTED = 9
path: modules/nextflow/src/test/groovy/nextflow/trace/ProgressRecordTest.groovy
user: login: jorgee
body: ```suggestion
        def ABORTED = 9
        def RETRIES = 1
```
created_at: 2025-05-06T13:45:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,30 +53,32 @@ class ProgressRecordTest extends Specification {
 
     def 'should get counts' () {
         given:
-        def PENDING =1
-        def SUBMITTED =2
-        def RUNNING =3
-        def SUCCEEDED =4
-        def FAILED =5
-        def CACHED =6
-        def STORED =7
-        def ABORTED = 8
+        def PENDING = 1
+        def SUBMITTED = 2
+        def RUNNING = 3
+        def SUCCEEDED = 4
+        def FAILED = 5
+        def IGNORED = 6
+        def CACHED = 7
+        def STORED = 8
+        def ABORTED = 9
         and:
         def rec = new ProgressRecord(10, 'foo')
 
         when:
-        rec.pending =PENDING
-        rec.submitted =SUBMITTED
-        rec.running =RUNNING
-        rec.succeeded =SUCCEEDED
-        rec.failed =FAILED
-        rec.cached =CACHED
-        rec.stored =STORED
+        rec.pending = PENDING
+        rec.submitted = SUBMITTED
+        rec.running = RUNNING
+        rec.succeeded = SUCCEEDED
+        rec.failed = FAILED
+        rec.ignored = IGNORED
+        rec.cached = CACHED
+        rec.stored = STORED
         rec.aborted = ABORTED
 
         then:
-        rec.getCompletedCount() == SUCCEEDED+ FAILED+ CACHED+ STORED
-        rec.getTotalCount() == PENDING+ SUBMITTED+ RUNNING + SUCCEEDED+ FAILED+ CACHED+ STORED+ ABORTED
+        rec.getCompletedCount() == SUCCEEDED + IGNORED + CACHED + STORED
+        rec.getTotalCount() == PENDING + SUBMITTED + RUNNING  + SUCCEEDED + IGNORED + CACHED + STORED + ABORTED
path: modules/nextflow/src/test/groovy/nextflow/trace/ProgressRecordTest.groovy
user: login: jorgee
body: ```suggestion
        rec.getTotalCount() == PENDING + SUBMITTED + RUNNING  + SUCCEEDED + IGNORED + FAILED - RETRIES + CACHED + STORED + ABORTED
```
created_at: 2025-05-06T13:45:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -61,13 +61,15 @@ class ProgressRecord implements Cloneable {
         this.taskName = processName
     }
 
+    // failed tasks that were not retried are included in the total count
     int getTotalCount() {
-        pending+ submitted+ running+
-           succeeded+ failed+ cached+ stored + aborted
+        pending + submitted + running +
+           succeeded + failed - retries + cached + stored + aborted
path: modules/nextflow/src/main/groovy/nextflow/trace/ProgressRecord.groovy
user: login: bentsherman
body: `failed` includes the ignored count here
created_at: 2025-05-08T13:13:56Z
author_association: MEMBER

PR comments:
user: login: netlify[bot]
created_at: 2025-03-10T17:06:53Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *nextflow-docs-staging* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 439cfacad63f6ad93b805282f76bd97d25e8933a |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/nextflow-docs-staging/deploys/681cb00509b5b200082a872f |
author_association: NONE
user: login: pditommaso
created_at: 2025-03-10T17:54:25Z
body: We need to make sure this is not going to impact Platform progress (also considering current redesign) 
author_association: MEMBER
user: login: bentsherman
created_at: 2025-03-10T18:09:06Z
body: It looks like getTotalCount() and getCompleteCount() are not sent to platform, in fact they are explicitly excluded:

https://github.com/nextflow-io/nextflow/blob/bb11d2e9f4aa8123790bb20cc58e13a6d64f7937/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerJsonGenerator.groovy#L61-L72

Which makes sense because platform needs the raw numbers anyway
author_association: MEMBER
user: login: jorgee
created_at: 2025-03-11T17:48:41Z
body: Reviewing the PR I have seen something similar is happening in the ReportObserver. I was testing if there was some side effect when running `-with-report` and I have seen it is reporting failed+ignored 9 tasks (3 ignored and 6 failed). 

![image](https://github.com/user-attachments/assets/6114095e-75b6-4839-bc67-0cee38c2c2f0)

To be coherent with platform report it should be reporting the failed. It is not a side effect of the PR, it is also happening in previous versions. Can we fix it in this PR or should I open a new issue?
author_association: CONTRIBUTOR
user: login: bentsherman
created_at: 2025-03-11T18:10:01Z
body: Let's fix it in this PR
author_association: MEMBER
user: login: jorgee
created_at: 2025-03-13T15:04:42Z
body: @bentsherman, I have pushed the fix for the report stats
author_association: CONTRIBUTOR
user: login: bentsherman
created_at: 2025-03-21T16:03:59Z
body: To fix the progress bar, I replaced the failed count with an "effective" failed count, which is just the failed minus the ignored/retried. So the percentages for succeeded + cached + ignored + "effective" failed should be correct
author_association: MEMBER
user: login: bentsherman
created_at: 2025-05-08T13:30:47Z
body: To summarize:

- Every failed task is either ignored, retried, or terminates the pipeline

- Retried tasks should not count towards the "total" count because a retry is essentially a duplicate of the first attempt

- Only ignored tasks should be counted towards the "completed" count because they are the only kind of failed task that represents progress

- In the execution report, retries should not be included in the progress bar, so we show the ignored count in yellow and the "effective failed" count (failed minus ignored/retried) in red
author_association: MEMBER
user: login: bentsherman
created_at: 2025-05-08T13:31:55Z
body: These changes have also been implemented in platform as part of the run detail enhancements for a coordinated release
author_association: MEMBER