Repository: DataBiosphere/toil

Issue:
ValueError: could not convert string to float: 
Hello,

I got a bug in the last few days when running cactus-prepare-toil. I think it's about the toil, so I am here. the bug is following:

```
Traceback (most recent call last):
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/bin/cactus-prepare-toil", line 8, in <module>
    sys.exit(main_toil())
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/cactus/progressive/cactus_prepare.py", line 46, in main_toil
    return main(toil_mode=True)
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/cactus/progressive/cactus_prepare.py", line 230, in main
    cactusPrepare(options, project)
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/cactus/progressive/cactus_prepare.py", line 393, in cactusPrepare
    toil.restart()
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/common.py", line 874, in restart
    return self._runMainLoop(rootJobDescription)
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/common.py", line 1132, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/leader.py", line 229, in run
    self.innerLoop()
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/leader.py", line 622, in innerLoop
    self.checkForDeadlocks()
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/leader.py", line 654, in checkForDeadlocks
    totalRunningJobs = len(self.batchSystem.getRunningBatchJobIDs())
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 371, in getRunningBatchJobIDs
    batchIds = self.with_retries(self.worker.getRunningJobIDs)
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 426, in with_retries
    return operation(*args, **kwargs)
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/batchSystems/torque.py", line 92, in getRunningJobIDs
    walltime = hms_to_seconds(walltime)
  File "/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/lib/conversions.py", line 100, in hms_to_seconds
    if(float(val) < 0): 
ValueError: could not convert string to float:
```

I think the bug is due to `/pwd/dragon_cactus/soft/cactus-bin-v2.0.2/venv/lib/python3.6/site-packages/toil/lib/conversions.py`, line 100. I google this bug, maybe the variable vals_to_convert(in line 96 of `conversions.py`) containing the data had some space (so not visible). This is just my guess, what do you think is causing this bug?

Yawen

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-962)
┆Issue Number: TOIL-962



Discussion:
I think maybe we just need to accept an empty time from the backing scheduler and treat it as 0.

Resolving patch:
diff --git a/src/toil/batchSystems/torque.py b/src/toil/batchSystems/torque.py
--- a/src/toil/batchSystems/torque.py
+++ b/src/toil/batchSystems/torque.py
@@ -91,14 +91,21 @@ def getRunningJobIDs(self):
                     if jobid in currentjobs:
                         logger.debug("getRunningJobIDs job status for is: %s", items[4])
                     if jobid in currentjobs and items[4] == 'R':
-                        walltime = items[3]
+                        walltime = items[3].strip()
                         logger.debug(
                             "getRunningJobIDs qstat reported walltime is: %s", walltime
                         )
                         # normal qstat has a quirk with job time where it reports '0'
                         # when initially running; this catches this case
                         if walltime == '0':
-                            walltime = time.mktime(time.strptime(walltime, "%S"))
+                            walltime = 0.0
+                        elif not walltime:
+                            # Sometimes we don't get any data here.
+                            # See https://github.com/DataBiosphere/toil/issues/3715
+                            logger.warning(
+                                "Assuming 0 walltime due to missing field in qstat line: %s", currline
+                            )
+                            walltime = 0.0
                         else:
                             walltime = hms_duration_to_seconds(walltime)
                         times[currentjobs[jobid]] = walltime


Test patch: