Repository: DataBiosphere/toil

Issue:
toil-cwl-runner works on single and fails on batch, mesos, aws
If I launch a cluster like so:

```
toil launch-cluster test \
    --leaderNodeType t2.xlarge \
    --keyPairName mykey \
    --zone us-east-1a \
    -p aws
```

and set it up properly then run `toil-cwl-runner`:

```
PRIVATE_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4); 
toil-cwl-runner \
--maxNodes 2 \
--nodeTypes t2.xlarge \
--debugWorker \
--provisioner aws \
--batchSystem mesos \
--mesosMaster ${PRIVATE_IP}:5050 \
--disableCaching \
--defaultMemory 6G \
--defaultCores 4 \
--logDebug \
--logFile batchSystem.nonpreempt.err.log \
/tmp/workflow.cwl \
/tmp/input.json
```

I will get a bunch of failures at, e.g.:

```
$grep Totally batchSystem.nonpreempt.err.log
Totally failed job: 'file:///tmp/qc/fastqc.cwl' fastqc B/X/jobaqaKIn is marking direct predecessor: 'CWLScatter' w/x/jobo1VwN5 as having failed jobs
Totally failed job: 'file:///tmp/qc/fastqc.cwl' fastqc p/9/job5rt616 is marking direct predecessor: 'CWLScatter' w/x/jobo1VwN5 as having failed jobs
Totally failed job: 'CWLScatter' w/x/jobo1VwN5 is marking direct predecessor: 'CWLWorkflow' 0/g/job4OpGO1 as having failed jobs
Totally failed job: 'file:///tmp/utils/extract-basename.cwl' echo S/e/jobDayC9g is marking direct predecessor: 'CWLScatter' 7/p/jobnZCDj2 as having failed jobs
Totally failed job: 'file:///tmp/utils/extract-basename.cwl' echo r/W/jobcb_RM_ is marking direct predecessor: 'CWLScatter' 7/p/jobnZCDj2 as having failed jobs
Totally failed job: 'CWLScatter' 7/p/jobnZCDj2 is marking direct predecessor: 'CWLWorkflow' 0/g/job4OpGO1 as having failed jobs
Totally failed job: 'CWLWorkflow' 0/g/job4OpGO1 is marking direct predecessor: 'CWLWorkflow' D/x/joboPYr0i as having failed jobs
```

It works on single machine mode, e.g.:

```
PRIVATE_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4); 
toil-cwl-runner \
--defaultMemory 6G \
--defaultCores 4 \
--logDebug \
--logFile singleMachine.nonpreempt.err.log \
/tmp/workflow.cwl \
/tmp/input.json
```
...so I suspect it has something to do with mesos communicating between the leader and worker nodes (see a snippet of `/var/lib/mesos/mesos-master.INFO` below). I don't think it has to do with my version of toil; I've tested several. Any ideas?

```
$tail -n 300  /var/lib/mesos/mesos-master.INFO
I0626 22:31:50.795078    13 master.cpp:4262] Processing ACKNOWLEDGE call e6968263-cba1-4327-82b5-e83cf353605a for task 7 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:50.795127    13 master.cpp:6907] Removing task 7 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:51.520443    13 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:31:51.521862     8 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O95 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
W0626 22:31:51.522172     9 validation.cpp:647] Executor toil-12911 for task 8 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:51.522224     9 validation.cpp:659] Executor toil-12911 for task 8 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:51.522276     9 master.cpp:7447] Adding task 8 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:51.522305     9 master.cpp:3831] Launching task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:51.527654    12 master.cpp:5137] Status update TASK_RUNNING (UUID: ebdc9074-e201-4d87-b0f1-a61b3bc0344d) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:51.527709    12 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: ebdc9074-e201-4d87-b0f1-a61b3bc0344d) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:31:51.527768    12 master.cpp:6841] Updating the state of task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:51.528162     8 master.cpp:4262] Processing ACKNOWLEDGE call ebdc9074-e201-4d87-b0f1-a61b3bc0344d for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:51.792137    11 master.cpp:5137] Status update TASK_FAILED (UUID: 3d9ba601-8324-4ce2-9f49-0962b29af777) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:51.792201    11 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 3d9ba601-8324-4ce2-9f49-0962b29af777) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:31:51.792289    11 master.cpp:6841] Updating the state of task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:51.793990    11 master.cpp:4262] Processing ACKNOWLEDGE call 3d9ba601-8324-4ce2-9f49-0962b29af777 for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:51.794030    11 master.cpp:6907] Removing task 8 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:52.521492     9 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:31:52.523118    14 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O96 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
W0626 22:31:52.523475    14 validation.cpp:647] Executor toil-14099 for task 8 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:52.523517    14 validation.cpp:659] Executor toil-14099 for task 8 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:52.523579    14 master.cpp:7447] Adding task 8 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:52.523622    14 master.cpp:3831] Launching task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:52.528591     7 master.cpp:5137] Status update TASK_RUNNING (UUID: 58668c03-78fd-4078-b199-0c96441b615f) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:52.528631     7 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 58668c03-78fd-4078-b199-0c96441b615f) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:31:52.528692     7 master.cpp:6841] Updating the state of task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:52.529135    12 master.cpp:4262] Processing ACKNOWLEDGE call 58668c03-78fd-4078-b199-0c96441b615f for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:52.805500    13 master.cpp:5137] Status update TASK_FAILED (UUID: 47ac2531-9c53-4dfb-8928-5b07eecf2e1e) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:52.805572    13 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 47ac2531-9c53-4dfb-8928-5b07eecf2e1e) for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:31:52.805656    13 master.cpp:6841] Updating the state of task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:52.806562    13 master.cpp:4262] Processing ACKNOWLEDGE call 47ac2531-9c53-4dfb-8928-5b07eecf2e1e for task 8 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:52.806593    13 master.cpp:6907] Removing task 8 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:53.522590    13 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:31:53.524847     9 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O97 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
W0626 22:31:53.525169     9 validation.cpp:647] Executor toil-14684 for task 9 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:53.525208     9 validation.cpp:659] Executor toil-14684 for task 9 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:53.525267     9 master.cpp:7447] Adding task 9 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:53.525300     9 master.cpp:3831] Launching task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.161686    10 master.cpp:5137] Status update TASK_RUNNING (UUID: 0684525d-115c-4b70-82dd-2e5f7b283be9) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.161747    10 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 0684525d-115c-4b70-82dd-2e5f7b283be9) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:31:54.161824    10 master.cpp:6841] Updating the state of task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:54.162847     9 master.cpp:4262] Processing ACKNOWLEDGE call 0684525d-115c-4b70-82dd-2e5f7b283be9 for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:54.433315    11 master.cpp:5137] Status update TASK_FAILED (UUID: 410a465b-4e79-4fa1-bb16-70e94f4e9d35) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.433377    11 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 410a465b-4e79-4fa1-bb16-70e94f4e9d35) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:31:54.433475    11 master.cpp:6841] Updating the state of task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:54.434567    11 master.cpp:4262] Processing ACKNOWLEDGE call 410a465b-4e79-4fa1-bb16-70e94f4e9d35 for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:54.434604    11 master.cpp:6907] Removing task 9 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.523368    14 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:31:54.524585    13 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O98 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
W0626 22:31:54.524875    12 validation.cpp:647] Executor toil-12911 for task 9 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:54.524919    12 validation.cpp:659] Executor toil-12911 for task 9 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:54.524972    12 master.cpp:7447] Adding task 9 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:54.525002    12 master.cpp:3831] Launching task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.528501    12 master.cpp:5137] Status update TASK_RUNNING (UUID: 4f8fbd42-56d1-43fe-ae76-956fd23fed13) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.528578    12 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 4f8fbd42-56d1-43fe-ae76-956fd23fed13) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:31:54.528656    12 master.cpp:6841] Updating the state of task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:54.529054     7 master.cpp:4262] Processing ACKNOWLEDGE call 4f8fbd42-56d1-43fe-ae76-956fd23fed13 for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:54.802307    10 master.cpp:5137] Status update TASK_FAILED (UUID: 250f5971-fcdd-4a0d-a0fd-391128229ad2) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:54.802372    10 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 250f5971-fcdd-4a0d-a0fd-391128229ad2) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:31:54.802451    10 master.cpp:6841] Updating the state of task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:54.803308    14 master.cpp:4262] Processing ACKNOWLEDGE call 250f5971-fcdd-4a0d-a0fd-391128229ad2 for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:54.803354    14 master.cpp:6907] Removing task 9 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:55.523836    11 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:31:55.525233     8 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O99 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
W0626 22:31:55.525537     8 validation.cpp:647] Executor toil-14099 for task 9 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:55.525574     8 validation.cpp:659] Executor toil-14099 for task 9 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:55.525619     8 master.cpp:7447] Adding task 9 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:55.525651     8 master.cpp:3831] Launching task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:55.529009    12 master.cpp:5137] Status update TASK_RUNNING (UUID: 06e99838-34f0-4839-bcae-d3928186c775) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:55.529049    12 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 06e99838-34f0-4839-bcae-d3928186c775) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:31:55.529114    12 master.cpp:6841] Updating the state of task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:55.529645    13 master.cpp:4262] Processing ACKNOWLEDGE call 06e99838-34f0-4839-bcae-d3928186c775 for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:55.794133    12 master.cpp:5137] Status update TASK_FAILED (UUID: df0f2d85-295c-4050-b895-1883309e27fe) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:55.794199    12 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: df0f2d85-295c-4050-b895-1883309e27fe) for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:31:55.794282    12 master.cpp:6841] Updating the state of task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:55.795107    12 master.cpp:4262] Processing ACKNOWLEDGE call df0f2d85-295c-4050-b895-1883309e27fe for task 9 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:55.795140    12 master.cpp:6907] Removing task 9 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:56.524451     7 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:31:56.526590     9 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O100 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
W0626 22:31:56.526921     8 validation.cpp:647] Executor toil-14684 for task 10 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:56.526958     8 validation.cpp:659] Executor toil-14684 for task 10 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:56.527007     8 master.cpp:7447] Adding task 10 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:56.527040     8 master.cpp:3831] Launching task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:56.530364    13 master.cpp:5137] Status update TASK_RUNNING (UUID: c4dd6bbe-714c-4ea3-80b9-698aa53a4982) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:56.530403    13 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: c4dd6bbe-714c-4ea3-80b9-698aa53a4982) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:31:56.530493    13 master.cpp:6841] Updating the state of task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:56.531092     7 master.cpp:4262] Processing ACKNOWLEDGE call c4dd6bbe-714c-4ea3-80b9-698aa53a4982 for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:56.792579    11 master.cpp:5137] Status update TASK_FAILED (UUID: 3e36cf95-5803-4400-a8c7-86b83ed7bdb9) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:56.792647    11 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 3e36cf95-5803-4400-a8c7-86b83ed7bdb9) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:31:56.793833    11 master.cpp:6841] Updating the state of task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:56.794903    11 master.cpp:4262] Processing ACKNOWLEDGE call 3e36cf95-5803-4400-a8c7-86b83ed7bdb9 for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:56.794951    11 master.cpp:6907] Removing task 10 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:57.525303     7 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:31:57.526738    14 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O101 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
W0626 22:31:57.527077    14 validation.cpp:647] Executor toil-12911 for task 10 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:57.527127    14 validation.cpp:659] Executor toil-12911 for task 10 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:57.527181    14 master.cpp:7447] Adding task 10 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:57.527207    14 master.cpp:3831] Launching task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:57.530628    10 master.cpp:5137] Status update TASK_RUNNING (UUID: 7b01f91f-9ee0-45d5-8051-f3d43639ed5f) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:57.530668    10 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 7b01f91f-9ee0-45d5-8051-f3d43639ed5f) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:31:57.530728    10 master.cpp:6841] Updating the state of task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:57.531220    11 master.cpp:4262] Processing ACKNOWLEDGE call 7b01f91f-9ee0-45d5-8051-f3d43639ed5f for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:57.799718    13 master.cpp:5137] Status update TASK_FAILED (UUID: 40a9f7a1-70ac-4c99-b7a2-de519ddecc61) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:57.799778    13 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 40a9f7a1-70ac-4c99-b7a2-de519ddecc61) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:31:57.799856    13 master.cpp:6841] Updating the state of task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:57.800765    13 master.cpp:4262] Processing ACKNOWLEDGE call 40a9f7a1-70ac-4c99-b7a2-de519ddecc61 for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:57.800809    13 master.cpp:6907] Removing task 10 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:58.527101     9 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:31:58.528570    12 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O102 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
W0626 22:31:58.528895    12 validation.cpp:647] Executor toil-14099 for task 10 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:58.528934    12 validation.cpp:659] Executor toil-14099 for task 10 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:58.528985    12 master.cpp:7447] Adding task 10 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:58.529017    12 master.cpp:3831] Launching task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:58.761663    14 master.cpp:5137] Status update TASK_RUNNING (UUID: 6e9c8209-3862-434e-83b2-d54a2ec7569e) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:58.761730    14 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 6e9c8209-3862-434e-83b2-d54a2ec7569e) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:31:58.761809    14 master.cpp:6841] Updating the state of task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:58.762395     8 master.cpp:4262] Processing ACKNOWLEDGE call 6e9c8209-3862-434e-83b2-d54a2ec7569e for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:59.025707    10 master.cpp:5137] Status update TASK_FAILED (UUID: 5216ac34-a993-44e6-b931-80cf1bd3c9b3) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:59.025785    10 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 5216ac34-a993-44e6-b931-80cf1bd3c9b3) for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:31:59.025873    10 master.cpp:6841] Updating the state of task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:59.026733    10 master.cpp:4262] Processing ACKNOWLEDGE call 5216ac34-a993-44e6-b931-80cf1bd3c9b3 for task 10 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:59.026799    10 master.cpp:6907] Removing task 10 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:59.528024    13 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:31:59.530069    12 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O103 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
W0626 22:31:59.530418    11 validation.cpp:647] Executor toil-14684 for task 11 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:31:59.530477    11 validation.cpp:659] Executor toil-14684 for task 11 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:31:59.530532    11 master.cpp:7447] Adding task 11 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:31:59.530561    11 master.cpp:3831] Launching task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:59.534299     9 master.cpp:5137] Status update TASK_RUNNING (UUID: 9329db4b-41bd-4407-9682-14829a59c867) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:59.534355     9 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 9329db4b-41bd-4407-9682-14829a59c867) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:31:59.534440     9 master.cpp:6841] Updating the state of task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:31:59.535118    11 master.cpp:4262] Processing ACKNOWLEDGE call 9329db4b-41bd-4407-9682-14829a59c867 for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:59.800079    12 master.cpp:5137] Status update TASK_FAILED (UUID: 71d381c1-cbd9-4171-949b-d88ec6135963) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:31:59.800138    12 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 71d381c1-cbd9-4171-949b-d88ec6135963) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:31:59.800210    12 master.cpp:6841] Updating the state of task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:31:59.801038    14 master.cpp:4262] Processing ACKNOWLEDGE call 71d381c1-cbd9-4171-949b-d88ec6135963 for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:31:59.801075    14 master.cpp:6907] Removing task 11 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:00.528620     7 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:00.530033    12 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O104 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
W0626 22:32:00.530405    12 validation.cpp:647] Executor toil-12911 for task 11 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:00.530442    12 validation.cpp:659] Executor toil-12911 for task 11 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:00.530488    12 master.cpp:7447] Adding task 11 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:00.530519    12 master.cpp:3831] Launching task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:00.533964    10 master.cpp:5137] Status update TASK_RUNNING (UUID: 53c5a880-e607-484a-8a15-186b1d67605d) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:00.534006    10 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 53c5a880-e607-484a-8a15-186b1d67605d) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:00.534090    10 master.cpp:6841] Updating the state of task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:00.534545    12 master.cpp:4262] Processing ACKNOWLEDGE call 53c5a880-e607-484a-8a15-186b1d67605d for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:00.809664     8 master.cpp:5137] Status update TASK_FAILED (UUID: 04ebc911-5fdb-4f63-81e3-769193ea81f7) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:00.809729     8 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 04ebc911-5fdb-4f63-81e3-769193ea81f7) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:00.809808     8 master.cpp:6841] Updating the state of task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:00.810798     8 master.cpp:4262] Processing ACKNOWLEDGE call 04ebc911-5fdb-4f63-81e3-769193ea81f7 for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:00.810837     8 master.cpp:6907] Removing task 11 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:01.529119     9 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:01.530541     8 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O105 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
W0626 22:32:01.530838    14 validation.cpp:647] Executor toil-14099 for task 11 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:01.530892    14 validation.cpp:659] Executor toil-14099 for task 11 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:01.530939    14 master.cpp:7447] Adding task 11 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:01.531024    14 master.cpp:3831] Launching task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:01.534318    14 master.cpp:5137] Status update TASK_RUNNING (UUID: 991528c0-665a-49c8-84dc-9270293aee5e) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:01.534354    14 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 991528c0-665a-49c8-84dc-9270293aee5e) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:01.534410    14 master.cpp:6841] Updating the state of task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:01.534932     8 master.cpp:4262] Processing ACKNOWLEDGE call 991528c0-665a-49c8-84dc-9270293aee5e for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:01.810964     7 master.cpp:5137] Status update TASK_FAILED (UUID: f8493216-0afc-46ff-9add-34f7048c1e75) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:01.811069     7 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: f8493216-0afc-46ff-9add-34f7048c1e75) for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:01.812394     7 master.cpp:6841] Updating the state of task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:01.813381     9 master.cpp:4262] Processing ACKNOWLEDGE call f8493216-0afc-46ff-9add-34f7048c1e75 for task 11 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:01.813416     9 master.cpp:6907] Removing task 11 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:02.529881    11 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:02.531982    10 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O106 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
W0626 22:32:02.532254    11 validation.cpp:647] Executor toil-14684 for task 12 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:02.532295    11 validation.cpp:659] Executor toil-14684 for task 12 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:02.532342    11 master.cpp:7447] Adding task 12 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:02.532407    11 master.cpp:3831] Launching task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:02.535979    12 master.cpp:5137] Status update TASK_RUNNING (UUID: e028fe9e-d27a-4724-b177-46be4f622700) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:02.536015    12 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: e028fe9e-d27a-4724-b177-46be4f622700) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:32:02.536072    12 master.cpp:6841] Updating the state of task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:02.536762    10 master.cpp:4262] Processing ACKNOWLEDGE call e028fe9e-d27a-4724-b177-46be4f622700 for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:02.800632    13 master.cpp:5137] Status update TASK_FAILED (UUID: 89a36222-7e20-4682-974b-8874fd3479d9) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:02.800701    13 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 89a36222-7e20-4682-974b-8874fd3479d9) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:32:02.800786    13 master.cpp:6841] Updating the state of task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:02.801798    13 master.cpp:4262] Processing ACKNOWLEDGE call 89a36222-7e20-4682-974b-8874fd3479d9 for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:02.801847    13 master.cpp:6907] Removing task 12 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:03.531199    11 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:03.532626     9 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O107 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
W0626 22:32:03.532974     9 validation.cpp:647] Executor toil-12911 for task 12 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:03.533015     9 validation.cpp:659] Executor toil-12911 for task 12 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:03.533077     9 master.cpp:7447] Adding task 12 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:03.533112     9 master.cpp:3831] Launching task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:03.556504     9 master.cpp:5137] Status update TASK_RUNNING (UUID: 52cc035f-4ba2-44b1-972d-7a2ca8065b2f) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:03.556560     9 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 52cc035f-4ba2-44b1-972d-7a2ca8065b2f) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:03.556648     9 master.cpp:6841] Updating the state of task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:03.557155    14 master.cpp:4262] Processing ACKNOWLEDGE call 52cc035f-4ba2-44b1-972d-7a2ca8065b2f for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:03.822358    10 master.cpp:5137] Status update TASK_FAILED (UUID: 5e530a65-806a-439e-8aab-7c97cd5bd31e) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:03.822424    10 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 5e530a65-806a-439e-8aab-7c97cd5bd31e) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:03.822520    10 master.cpp:6841] Updating the state of task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:03.823575    10 master.cpp:4262] Processing ACKNOWLEDGE call 5e530a65-806a-439e-8aab-7c97cd5bd31e for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:03.823609    10 master.cpp:6907] Removing task 12 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:04.531894    13 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:04.533373     7 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O108 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
W0626 22:32:04.533709     7 validation.cpp:647] Executor toil-14099 for task 12 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:04.533749     7 validation.cpp:659] Executor toil-14099 for task 12 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:04.533795     7 master.cpp:7447] Adding task 12 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:04.533830     7 master.cpp:3831] Launching task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:04.537106    11 master.cpp:5137] Status update TASK_RUNNING (UUID: e3086ea5-66a8-4f36-a229-b14bd281dd57) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:04.537144    11 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: e3086ea5-66a8-4f36-a229-b14bd281dd57) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:04.537212    11 master.cpp:6841] Updating the state of task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:04.537705    12 master.cpp:4262] Processing ACKNOWLEDGE call e3086ea5-66a8-4f36-a229-b14bd281dd57 for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:04.811148     9 master.cpp:5137] Status update TASK_FAILED (UUID: 40cd9e70-f8ad-400e-917d-9a649bfc237d) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:04.811215     9 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 40cd9e70-f8ad-400e-917d-9a649bfc237d) for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:04.811297     9 master.cpp:6841] Updating the state of task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:04.812240     8 master.cpp:4262] Processing ACKNOWLEDGE call 40cd9e70-f8ad-400e-917d-9a649bfc237d for task 12 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:04.812294     8 master.cpp:6907] Removing task 12 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:05.532755    12 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:05.534744     9 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O109 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
W0626 22:32:05.535084    11 validation.cpp:647] Executor toil-14684 for task 13 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:05.535127    11 validation.cpp:659] Executor toil-14684 for task 13 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:05.535172    11 master.cpp:7447] Adding task 13 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:05.535205    11 master.cpp:3831] Launching task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:05.538354    12 master.cpp:5137] Status update TASK_RUNNING (UUID: eb4c0aa1-a977-4ecb-9629-0e7c4f989973) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:05.538404    12 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: eb4c0aa1-a977-4ecb-9629-0e7c4f989973) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:32:05.538489    12 master.cpp:6841] Updating the state of task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:05.539176     7 master.cpp:4262] Processing ACKNOWLEDGE call eb4c0aa1-a977-4ecb-9629-0e7c4f989973 for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:05.808760    14 master.cpp:5137] Status update TASK_FAILED (UUID: 3750d571-0c37-45b1-b505-8350e6761ff3) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:05.808826    14 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 3750d571-0c37-45b1-b505-8350e6761ff3) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:32:05.808908    14 master.cpp:6841] Updating the state of task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:05.809890     8 master.cpp:4262] Processing ACKNOWLEDGE call 3750d571-0c37-45b1-b505-8350e6761ff3 for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:05.809931     8 master.cpp:6907] Removing task 13 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:06.533241    12 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:06.534837    14 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O110 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
W0626 22:32:06.535213    13 validation.cpp:647] Executor toil-12911 for task 13 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:06.535281    13 validation.cpp:659] Executor toil-12911 for task 13 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:06.535352    13 master.cpp:7447] Adding task 13 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:06.535462    13 master.cpp:3831] Launching task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:06.539036    14 master.cpp:5137] Status update TASK_RUNNING (UUID: a46445c2-f7f6-4c4b-b604-abb08c820169) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:06.539072    14 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: a46445c2-f7f6-4c4b-b604-abb08c820169) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:06.539129    14 master.cpp:6841] Updating the state of task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:06.539602    13 master.cpp:4262] Processing ACKNOWLEDGE call a46445c2-f7f6-4c4b-b604-abb08c820169 for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:06.802899     7 master.cpp:5137] Status update TASK_FAILED (UUID: 31a5b57a-0d74-4f3b-8393-c7e316ca1093) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:06.802964     7 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 31a5b57a-0d74-4f3b-8393-c7e316ca1093) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:06.803061     7 master.cpp:6841] Updating the state of task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:06.804699     9 master.cpp:4262] Processing ACKNOWLEDGE call 31a5b57a-0d74-4f3b-8393-c7e316ca1093 for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:06.804729     9 master.cpp:6907] Removing task 13 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:07.534095     7 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:07.535529    11 master.cpp:3342] Processing ACCEPT call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O111 ] on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
W0626 22:32:07.535827    10 validation.cpp:647] Executor toil-14099 for task 13 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0626 22:32:07.535871    10 validation.cpp:659] Executor toil-14099 for task 13 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0626 22:32:07.535918    10 master.cpp:7447] Adding task 13 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0)
I0626 22:32:07.535950    10 master.cpp:3831] Launching task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 with resources cpus(*):4; disk(*):3072; mem(*):6144 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:07.539026    11 master.cpp:5137] Status update TASK_RUNNING (UUID: 3dbf6158-a1e9-46b8-8c94-e9732fc31185) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:07.539093    11 master.cpp:5199] Forwarding status update TASK_RUNNING (UUID: 3dbf6158-a1e9-46b8-8c94-e9732fc31185) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:07.539172    11 master.cpp:6841] Updating the state of task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0626 22:32:07.539644    13 master.cpp:4262] Processing ACKNOWLEDGE call 3dbf6158-a1e9-46b8-8c94-e9732fc31185 for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:07.812116    14 master.cpp:5137] Status update TASK_FAILED (UUID: 67e4e43a-4c4c-46bd-83b6-66266df47cfb) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 from agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:07.812180    14 master.cpp:5199] Forwarding status update TASK_FAILED (UUID: 67e4e43a-4c4c-46bd-83b6-66266df47cfb) for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:07.812261    14 master.cpp:6841] Updating the state of task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I0626 22:32:07.813127    10 master.cpp:4262] Processing ACKNOWLEDGE call 67e4e43a-4c4c-46bd-83b6-66266df47cfb for task 13 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:32:07.813179    10 master.cpp:6907] Removing task 13 with resources cpus(*):4; disk(*):3072; mem(*):6144 of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:08.535006    10 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:08.536188    12 master.cpp:3951] Processing DECLINE call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O112 ] for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:09.535593    10 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:09.536433    14 master.cpp:3951] Processing DECLINE call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O113 ] for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:10.536875    12 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:10.537752    14 master.cpp:3951] Processing DECLINE call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O114 ] for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:13.539727    12 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:13.540956    10 master.cpp:3951] Processing DECLINE call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O115 ] for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:14.540504    13 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:14.541363    10 master.cpp:3951] Processing DECLINE call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O116 ] for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:15.540899    10 master.cpp:5725] Sending 1 offers to framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:15.541744    14 master.cpp:3951] Processing DECLINE call for offers: [ 8633f6fa-8078-426f-925e-238112970aa9-O117 ] for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:15.588315     9 master.cpp:6300] Processing TEARDOWN call for framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:15.588356     9 master.cpp:6312] Removing framework 8633f6fa-8078-426f-925e-238112970aa9-0009 (toil) at scheduler-c7062df3-71d6-4a28-bbfc-c00ee379e163@172.31.40.171:37417
I0626 22:32:15.588429     9 master.cpp:6936] Removing executor 'toil-12911' with resources  of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:15.588430    11 hierarchical.cpp:382] Deactivated framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:15.588527    11 hierarchical.cpp:333] Removed framework 8633f6fa-8078-426f-925e-238112970aa9-0009
I0626 22:32:17.043633    14 master.cpp:6300] Processing TEARDOWN call for framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:17.043766    14 master.cpp:6312] Removing framework 8633f6fa-8078-426f-925e-238112970aa9-0010 (toil) at scheduler-5483427f-67c1-4ea3-abef-c11926656f46@172.31.40.171:46775
I0626 22:32:17.043859     7 hierarchical.cpp:382] Deactivated framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:17.043896    14 master.cpp:6936] Removing executor 'toil-14099' with resources  of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:17.044049    14 hierarchical.cpp:333] Removed framework 8633f6fa-8078-426f-925e-238112970aa9-0010
I0626 22:32:17.944278     7 master.cpp:6300] Processing TEARDOWN call for framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:17.944321     7 master.cpp:6312] Removing framework 8633f6fa-8078-426f-925e-238112970aa9-0012 (toil) at scheduler-575ae254-9fc2-45db-91fc-97591bffb28c@172.31.40.171:36491
I0626 22:32:17.944430     7 master.cpp:6936] Removing executor 'toil-14684' with resources  of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:32:17.944562     7 hierarchical.cpp:382] Deactivated framework 8633f6fa-8078-426f-925e-238112970aa9-0012
I0626 22:32:17.944636     7 hierarchical.cpp:333] Removed framework 8633f6fa-8078-426f-925e-238112970aa9-0012
W0626 22:32:20.617219    13 master.cpp:5259] Ignoring unknown exited executor 'toil-12911' of framework 8633f6fa-8078-426f-925e-238112970aa9-0009 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
W0626 22:32:22.122926    11 master.cpp:5259] Ignoring unknown exited executor 'toil-14099' of framework 8633f6fa-8078-426f-925e-238112970aa9-0010 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
W0626 22:32:23.027761    10 master.cpp:5259] Ignoring unknown exited executor 'toil-14684' of framework 8633f6fa-8078-426f-925e-238112970aa9-0012 on agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:33:57.889103     9 master.cpp:1245] Agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) disconnected
I0626 22:33:57.889169     9 master.cpp:2784] Disconnecting agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:33:57.889204     9 master.cpp:2803] Deactivating agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0)
I0626 22:33:57.889238     9 hierarchical.cpp:571] Agent 8633f6fa-8078-426f-925e-238112970aa9-S6 deactivated
E0626 22:35:15.008669    15 process.cpp:2105] Failed to shutdown socket with fd 9: Transport endpoint is not connected
I0626 22:35:23.141489    11 master.cpp:244] Shutting down agent 8633f6fa-8078-426f-925e-238112970aa9-S6 due to health check timeout
W0626 22:35:23.141564    11 master.cpp:5331] Shutting down agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0) with message 'health check timed out'
I0626 22:35:23.141602    11 master.cpp:6614] Removing agent 8633f6fa-8078-426f-925e-238112970aa9-S6 at slave(1)@172.31.39.0:5051 (172.31.39.0): health check timed out
I0626 22:35:23.141732    13 hierarchical.cpp:510] Removed agent 8633f6fa-8078-426f-925e-238112970aa9-S6
I0626 22:35:23.141798    14 registrar.cpp:464] Applied 1 operations in 20920ns; attempting to update the 'registry'
I0626 22:35:23.141993    14 registrar.cpp:509] Successfully updated the 'registry' in 155904ns
I0626 22:35:23.142050    14 master.cpp:6733] Removed agent 8633f6fa-8078-426f-925e-238112970aa9-S6 (172.31.39.0): health check timed out
E0626 22:35:26.208745    15 process.cpp:2105] Failed to shutdown socket with fd 9: Transport endpoint is not connected
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-289)
┆Issue Number: TOIL-289



Discussion:
You can close this. After logging onto a worker node and examining `/var/lib/mesos/slaves/*/frameworks/*/executors/toil-*/runs/latest/stderr`, it became obvious that the workers did not have access to my jobStore. Adding `--jobStore aws:us-east-1:job-store-name` to the `toil-cwl-runner` call fixed the problem.

For others who run into this issue, it would have been helpful for the `NoSuchJobStoreException` to be passed from the worker back to the leader. Also, a brief mention of a batch run, with `--jobStore` included, may be helpful in http://toil.readthedocs.io/en/3.15.0/gettingStarted/quickStart.html#running-a-cwl-workflow-on-aws. 

I think adding a `NoSuchJobStoreException` when this occurs in the future would be useful, so I don't mind keeping this open til that's addressed.

Glad it was resolved!

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -50,6 +50,7 @@
 import cwltool.stdfsaccess
 import cwltool.command_line_tool
 
+from toil.jobStores.abstractJobStore import NoSuchJobStoreException
 from cwltool.loghandler import _logger as cwllogger
 from cwltool.loghandler import defaultStreamHandler
 from cwltool.pathmapper import (PathMapper, adjustDirObjs, adjustFileObjs,
@@ -1062,6 +1063,10 @@ def main(args=None, stdout=sys.stdout):
     # we use workdir as jobStore:
     options = parser.parse_args([workdir] + args)
 
+    if options.provisioner and not options.jobStore:
+        raise NoSuchJobStoreException(
+            'Please specify a jobstore with the --jobStore option when specifying a provisioner.')
+
     use_container = not options.no_container
 
     if options.logLevel:


Test patch: