Repository: DataBiosphere/toil

Issue:
Timestamps are not set in the Mesos dashboard
Timestamps are not set in the Mesos dashboard (localhost:5050) when trying the CWL runner on AWS. It shows "49 years ago" (aka epoch) for everything.

I think it's caused by [this commit](https://github.com/DataBiosphere/toil/commit/8efb7cabcc04b3b052cc1d8a1319ca19d260d723) as the build from the previous commit (just after the 3.18.0 release) works fine, but i haven't figured out the root cause yet.

@adamnovak FYI, in case you have any ideas.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-51)
┆Issue Number: TOIL-51



Discussion:
Yeah, I noticed something like this when originally working on the
Pymesos stuff. I guess I forgot to fix it.

Toil is using *relative* timestamps on everything it is feeding to
Mesos. It needs to be changed to use *absolute* timestamps, so that
Mesos doesn't think we started doing everything at midnight on Jan 1
1970.

We need to find every timestamp that our Mesos-using code sends, and convert it.

On 2/25/19, Asha Rostamianfar <notifications@github.com> wrote:
> Timestamps are not set in the Mesos dashboard (localhost:5050) when trying
> the CWL runner on AWS. It shows "49 years ago" (aka epoch) for everything.
>
> I think it's caused by [this
> commit](https://github.com/DataBiosphere/toil/commit/8efb7cabcc04b3b052cc1d8a1319ca19d260d723)
> as the build from the previous commit (just after the 3.18.0 release) works
> fine, but i haven't figured out the root cause yet.
>
> @adamnovak FYI, in case you have any ideas.
>
>
>
> --
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub:
> https://github.com/DataBiosphere/toil/issues/2526


Resolving patch:
diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -623,8 +623,14 @@ def jobEnded(_exitStatus, wallTime=None):
 
         if update.state == 'TASK_FINISHED':
             # We get the running time of the job via the timestamp, which is in job-local time in seconds
-            assert(update.has_key('timestamp'))
-            jobEnded(0, wallTime=update.timestamp)
+            labels = update.labels.labels
+            wallTime = None
+            for label in labels:
+                if label['key'] == 'wallTime':
+                    wallTime = float(label['value'])
+                    break
+            assert(wallTime is not None)
+            jobEnded(0, wallTime=wallTime)
         elif update.state == 'TASK_FAILED':
             try:
                 exitStatus = int(update.message)
diff --git a/src/toil/batchSystems/mesos/executor.py b/src/toil/batchSystems/mesos/executor.py
--- a/src/toil/batchSystems/mesos/executor.py
+++ b/src/toil/batchSystems/mesos/executor.py
@@ -212,15 +212,20 @@ def runJob(job):
                                         preexec_fn=lambda: os.setpgrp(),
                                         shell=True, env=jobEnv)
 
-        def sendUpdate(task, taskState, wallTime=None, msg=''):
+        def sendUpdate(task, taskState, wallTime, msg=''):
             update = addict.Dict()
             update.task_id.value = task.task_id.value
             if self.id is not None:
                 # Sign our messages as from us, since the driver doesn't do it.
                 update.executor_id.value = self.id
             update.state = taskState
-            update.timestamp = wallTime
             update.message = msg
+
+            # Add wallTime as a label.
+            labels = addict.Dict()
+            labels.labels = [{'key': 'wallTime', 'value': str(wallTime)}]
+            update.labels = labels
+
             driver.sendStatusUpdate(update)
 
         thread = threading.Thread(target=runTask)


Test patch: