Repository: DataBiosphere/toil

Issue:
Logging of stderr stdout disappears from worker log in toil 6.x 
I've bumped toil from v5.12.0 to v6.1.0a1, and it seems that the worker log no longer contains useful stdout/stderr info, but only other logging info that wasn't helpful in root cause. Running local seems fine, I can see the errors, but when running on cluster aws, the log wasn't there. Below was an example with purposely an error in a python script,

**In v5.12.0**, the command was 
```
toil-cwl-runner --provisioner aws --batchSystem mesos --destBucket s3://xx --jobStore xx --clean never --nodeStorage 500 --nodeTypes t3.medium:2,t3.large:2,t3.xlarge:2,c5.xlarge:2,c5.2xlarge:2,c5.4xlarge:2,r5.large:2,r5.xlarge:2,r5.2xlarge:2,r5.4xlarge:2,r5.8xlarge:3,r5d.xlarge:2,r5d.2xlarge:2,r5d.4xlarge:2,r5d.8xlarge:3,c5d.large:2,c5d.2xlarge:2,c5d.4xlarge:3 --maxNodes 15 --retryCount 2 --defaultPreemptible True --preemptibleCompensation 0.2 --logFile /tmp/logfile.txt --maxLogFileSize 100000000 --writeLogs /logs/ --symlinkImports True --no-read-only --rotatingLogging --logDebug /tmp/helloworld_error.cwl /tmp/job.yaml
```

and the worker log looks like
```
	[2024-02-10T10:55:27+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2024-02-10T10:55:27+0000] [MainThread] [I] [toil] Running Toil version 5.12.0-6d5a5b83b649cd8adf34a5cfe89e7690c95189d3 on host ip-172-31-38-173.
	[2024-02-10T10:55:27+0000] [MainThread] [D] [toil] Configuration: {'workflowID': '8c56c9ef-e0c6-4bf3-8b0b-a9d6253b9a59', 'workflowAttemptNumber': 0, 'jobStore': 'xxx', 'logLevel': 'DEBUG', 'workDir': None, 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'clusterStats': None, 'restart': False, 'batchSystem': 'mesos', 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': 2, 'manualMemArgs': False, 'statePollingWait': None, 'aws_batch_region': None, 'aws_batch_queue': None, 'aws_batch_job_role_arn': None, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1.0, 'mesos_endpoint': '172.31.43.220:5050', 'mesos_name': 'toil', 'mesos_role': None, 'mesos_framework_id': None, 'allocate_mem': False, 'tes_endpoint': 'http://172.31.43.220:8000', 'tes_user': None, 'tes_password': None, 'tes_bearer_token': None, 'kubernetes_host_path': None, 'kubernetes_owner': 'root', 'kubernetes_service_account': None, 'kubernetes_pod_timeout': 120, 'run_local_jobs_on_workers': False, 'caching': True, 'linkImports': True, 'moveExports': False, 'provisioner': 'aws', 'nodeTypes': [({'t3.medium'}, 2.0), ({'t3.large'}, 2.0), ({'t3.xlarge'}, 2.0), ({'c5.xlarge'}, 2.0), ({'c5.2xlarge'}, 2.0), ({'c5.4xlarge'}, 2.0), ({'r5.large'}, 2.0), ({'r5.xlarge'}, 2.0), ({'r5.2xlarge'}, 2.0), ({'r5.4xlarge'}, 2.0), ({'r5.8xlarge'}, 3.0), ({'r5d.xlarge'}, 2.0), ({'r5d.2xlarge'}, 2.0), ({'r5d.4xlarge'}, 2.0), ({'r5d.8xlarge'}, 3.0), ({'c5d.large'}, 2.0), ({'c5d.2xlarge'}, 2.0), ({'c5d.4xlarge'}, 3.0)], 'minNodes': None, 'maxNodes': [15], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptibleCompensation': 0.2, 'nodeStorage': 500, 'nodeStorageOverrides': [], 'metrics': False, 'assume_zero_overhead': False, 'maxPreemptibleServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultPreemptible': True, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 2, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 100000000, 'writeLogs': '/logs/', 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': '/app/tmp/tmps2vz6b7p', 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'statusWait': 3600, 'disableProgress': False, 'readGlobalFileMutableByDefault': False, 'kill_polling_interval': 5, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False, 'cleanWorkDir': 'always'}
...
	    /bin/sh \
	    -c \
	    python helloworld.py hello_world_msg > /var/lib/toil/9303913b73d8566aa84a53240af6ef61/d0d9/37e8/tmp1ihennti/tmp-out0ohwlx2h/helloworld.txt
	Traceback (most recent call last):
	  File "helloworld.py", line 5, in <module>
	    txt.str.split(',')
	AttributeError: 'str' object has no attribute 'str'
	[2024-02-10T10:55:29+0000] [MainThread] [I] [cwltool] [job helloworld] Max memory used: 0MiB
	[2024-02-10T10:55:29+0000] [MainThread] [W] [cwltool] [job helloworld] exited with status: 1
	[2024-02-10T10:55:29+0000] [MainThread] [W] [cwltool] [job helloworld] completed permanentFail
...
```


**In v6.1.0a1**, and command was
```
toil-cwl-runner --provisioner aws --batchSystem mesos --destBucket xx --jobStore xx --clean onSuccess --nodeStorage 500 --nodeTypes t3.medium:2,t3.large:2,t3.xlarge:2,c5.xlarge:2,c5.2xlarge:2,c5.4xlarge:2,r5.large:2,r5.xlarge:2,r5.2xlarge:2,r5.4xlarge:2,r5.8xlarge:3,r5d.xlarge:2,r5d.2xlarge:2,r5d.4xlarge:2,r5d.8xlarge:3,c5d.large:2,c5d.2xlarge:2,c5d.4xlarge:3 --maxNodes 15 --retryCount 2 --defaultPreemptable True --preemptableCompensation 0.2 --logFile /tmp/logfile.txt --maxLogFileSize 100000000 --writeLogs /logs/ --no-read-only --linkImports --rotatingLogging --logDebug /tmp/helloworld_error.cwl /tmp/job.yaml
```

and the worker log was,
```
	[2024-02-10T10:26:44+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2024-02-10T10:26:44+0000] [MainThread] [I] [toil] Running Toil version 6.1.0a1-73f155efd71ace58df5e54b0bc6ccbe9e9422f25 on host ip-172-31-40-224.
	  [2024-02-10T10:26:44+0000] [MainThread] [D] [toil] Configuration: {'cwl': False, 'workflowID': '986f4446-d28b-4c9e-bb2b-5590ff2eddf2', 'kill_polling_interval': 5, 'jobStore': 'xxx', 'workDir': '/var/lib/toil', 'coordination_dir': None, 'noStdOutErr': False, 'stats': False, 'cleanWorkDir': 'always', 'clean': 'never', 'clusterStats': None, 'restart': False, 'batchSystem': 'mesos', 'aws_batch_region': None, 'aws_batch_queue': None, 'aws_batch_job_role_arn': None, 'scale': 1, 'mesos_endpoint': None, 'mesos_name': 'toil', 'mesos_role': None, 'mesos_framework_id': None, 'allocate_mem': True, 'kubernetes_host_path': None, 'kubernetes_owner': None, 'kubernetes_service_account': None, 'kubernetes_pod_timeout': 120, 'disableAutoDeployment': False, 'max_jobs': 9223372036854775807, 'max_local_jobs': None, 'manualMemArgs': False, 'run_local_jobs_on_workers': False, 'statePollingWait': None, 'batch_logs_dir': None, 'symlinkImports': True, 'moveOutputs': False, 'caching': True, 'provisioner': 'aws', 'nodeTypes': [({'t3.medium'}, 2.0), ({'t3.large'}, 2.0), ({'t3.xlarge'}, 2.0), ({'c5.xlarge'}, 2.0), ({'c5.2xlarge'}, 2.0), ({'c5.4xlarge'}, 2.0), ({'r5.large'}, 2.0), ({'r5.xlarge'}, 2.0), ({'r5.2xlarge'}, 2.0), ({'r5.4xlarge'}, 2.0), ({'r5.8xlarge'}, 3.0), ({'r5d.xlarge'}, 2.0), ({'r5d.2xlarge'}, 2.0), ({'r5d.4xlarge'}, 2.0), ({'r5d.8xlarge'}, 3.0), ({'c5d.large'}, 2.0), ({'c5d.2xlarge'}, 2.0), ({'c5d.4xlarge'}, 3.0)], 'minNodes': [0], 'maxNodes': [15], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'metrics': False, 'assume_zero_overhead': False, 'preemptibleCompensation': 0.2, 'nodeStorage': 500, 'nodeStorageOverrides': [], 'maxServiceJobs': 9223372036854775807, 'maxPreemptibleServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'defaultAccelerators': [], 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'defaultPreemptible': True, 'retryCount': 2, 'enableUnlimitedPreemptibleRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 60, 'maxLogFileSize': 100000000, 'writeLogs': '/logs/', 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'write_messages': None, 'environment': {}, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'statusWait': 3600, 'disableProgress': False, 'sseKey': None, 'servicePollingInterval': 60.0, 'forceDockerAppliance': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'logLevel': 'Debug', 'workflowAttemptNumber': 0}
...
	    /bin/sh \
	    -c \
	    python helloworld.py hello_world_msg > /var/lib/toil/toilwf-c227a203b7495b1e970c1694d82f2f5b/a12e/04a1/tmppdyz8hqg/tmp-out1vr46gv0/helloworld.txt
	[2024-02-10T10:26:46+0000] [MainThread] [W] [cwltool] [job helloworld] exited with status: 1
	[2024-02-10T10:26:46+0000] [MainThread] [W] [cwltool] [job helloworld] completed permanentFail
...
```

Note the `Traceback (most recent call last):` text was missing in the run with v6.1.0a. Note sure if the Debug param in lower case was the issue.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1497)
┆Issue Number: TOIL-1497



Discussion:
Instead of being logged in the worker log, task standard output and standard error is now meant to be logged separately, under the heading (for this example) `helloworld.stdout`/`helloworld.stderr`, without needing `--logDebug`.

But it could be that it's not actually getting printed because the log reporting thread isn't managing to find and print it before the whole workflow stops due to the job failure?

I changed the WDL implementation to also reproduce the task log inline in the worker log on failure; maybe the CWL implementation also needs to do that.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -2620,13 +2620,34 @@ def run(self, file_store: AbstractFileStore) -> Any:
         logger.debug("Running tool %s with order: %s", self.cwltool, self.cwljob)
 
         runtime_context.name = self.description.unitName
-        output, status = ToilSingleJobExecutor().execute(
-            process=self.cwltool,
-            job_order_object=cwljob,
-            runtime_context=runtime_context,
-            logger=cwllogger,
-        )
-        ended_at = datetime.datetime.now()  # noqa F841
+
+        status = "did_not_run"
+        try:
+            output, status = ToilSingleJobExecutor().execute(
+                process=self.cwltool,
+                job_order_object=cwljob,
+                runtime_context=runtime_context,
+                logger=cwllogger,
+            )
+        finally:
+            ended_at = datetime.datetime.now()  # noqa F841
+
+            # Log any output/error data
+            default_stdout.seek(0, os.SEEK_END)
+            if default_stdout.tell() > 0:
+                default_stdout.seek(0)
+                file_store.log_user_stream(self.description.unitName + '.stdout', default_stdout)
+                if status != "success":
+                    default_stdout.seek(0)
+                    logger.error("Failed command standard output:\n%s", default_stdout.read().decode("utf-8", errors="replace"))
+            default_stderr.seek(0, os.SEEK_END)
+            if default_stderr.tell():
+                default_stderr.seek(0)
+                file_store.log_user_stream(self.description.unitName + '.stderr', default_stderr)
+                if status != "success":
+                    default_stderr.seek(0)
+                    logger.error("Failed command standard error:\n%s", default_stderr.read().decode("utf-8", errors="replace"))
+
         if status != "success":
             raise cwl_utils.errors.WorkflowException(status)
 
@@ -2634,16 +2655,6 @@ def run(self, file_store: AbstractFileStore) -> Any:
             os.close(fd)
             t.join()
 
-        # Log any output/error data
-        default_stdout.seek(0, os.SEEK_END)
-        if default_stdout.tell() > 0:
-            default_stdout.seek(0)
-            file_store.log_user_stream(self.description.unitName + '.stdout', default_stdout)
-        default_stderr.seek(0, os.SEEK_END)
-        if default_stderr.tell():
-            default_stderr.seek(0)
-            file_store.log_user_stream(self.description.unitName + '.stderr', default_stderr)
-
         # Get ahold of the filesystem
         fs_access = runtime_context.make_fs_access(runtime_context.basedir)
 


Test patch: