anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: hyperledger/cello
Issue: Kubernetes agent create node, when check deployment is alive if pod is not created will cause crash

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
<!--- Tell us what should happen -->
Can update node status after node is deployed.

## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
After node is deployed, cause crashed when check node status.

```bash
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    create_node()
  File "/app/operations/create_node.py", line 162, in create_node
    _create_fabric_node()
  File "/app/operations/create_node.py", line 140, in _create_fabric_node
    print(pod, pod.status)
AttributeError: 'NoneType' object has no attribute 'status'
```

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
Check if pod is not null object at first.

## Steps to Reproduce
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1. make start
2. MOCK=False PROXY=http://9.59.196.20:8085/engine make start-dashboard-dev `run at your computer`
3. create new agent
4. apply agent
5. create new node

## Context (Environment)
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

<!--- Provide a general summary of the issue in the Title above -->
OS: Ubuntu 18.04
Arch: x86_64
Kubernetes version: 1.13
Docker version: 19.03.3
Docker compose version: 1.24.1

## Detailed Description
<!--- Provide a detailed description of the change or addition you are proposing -->

## Possible Implementation
<!--- Not obligatory, but suggest an idea for implementing addition or change -->


Changed files:
src/agent/kubernetes-agent/src/operations/create_node.py

diff --git a/src/agent/kubernetes-agent/src/operations/create_node.py b/src/agent/kubernetes-agent/src/operations/create_node.py
index e338524b..4cfd4192 100644
--- a/src/agent/kubernetes-agent/src/operations/create_node.py
+++ b/src/agent/kubernetes-agent/src/operations/create_node.py
@@ -137,7 +137,7 @@ def _create_fabric_node():
     node_status = NodeStatus.Error.value
     for i in range(1, MAX_QUERY_RETRY):
         pod = k8s_client.get_pod(AGENT_ID, deploy_name)
-        if pod.status.phase == "Running":
+        if pod and pod.status.phase == "Running":
             node_status = NodeStatus.Running.value
             break
         sleep(5)
Repository: databiosphere/toil
Issue: logToMaster() doesn't work

This is an example from the docs. It does not print the logToMaster() message, neither on master nor on 3.1.x.

```
from toil.job import Job

class HelloWorld(Job):
    def __init__(self, message):
        Job.__init__(self,  memory="2G", cores=2, disk="3G")
        self.message = message

    def run(self, fileStore):
        fileStore.logToMaster("Hello, world!, I have a message: %s"
                              % self.message)

if __name__=="__main__":
    options = Job.Runner.getDefaultOptions("./toilWorkflowRun")
    options.logLevel = "INFO"
    Job.Runner.startToil(HelloWorld("woot"), options)
```


Changed files:
src/toil/leader.py, src/toil/worker.py

diff --git a/src/toil/leader.py b/src/toil/leader.py
index 0fd8682b..5f30dd36 100644
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -45,7 +45,7 @@ def statsAndLoggingAggregatorProcess(jobStore, stop):
     def callback(fileHandle):
         stats = json.load(fileHandle, object_hook=Expando)
         try:
-            logs = stats.workers.log
+            logs = stats.workers.logsToMaster
         except AttributeError:
             # To be expected if there were no calls to logToMaster()
             pass
diff --git a/src/toil/worker.py b/src/toil/worker.py
index 46dd1d63..ac38a6fb 100644
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -392,8 +392,9 @@ def main():
             statsDict.workers.time = str(time.time() - startTime)
             statsDict.workers.clock = str(totalCPUTime - startClock)
             statsDict.workers.memory = str(totalMemoryUsage)
-            statsDict.workers.log = messages
-        
+
+        statsDict.workers.logsToMaster = messages  # logToMaster messages should be always be passed
+
         logger.info("Finished running the chain of jobs on this node, we ran for a total of %f seconds", time.time() - startTime)
     
     ##########################################
Repository: databiosphere/toil
Issue: Global name 'config' not defined for getting azure job store

`global name 'config' is not defined` error found in `toil/common.py` on line 617 in `getJobStore` when running toil version 3.3.1 on an Azure cluster. Error does not show up for toil version 3.2.1 and the toil script runs to completion. File `.toilAzureCredentials` is set properly.

Toil script command:
`./vg_evaluation_pipeline.py --batchSystem mesos --mesosMaster 10.0.0.5:5050 --realTimeLogging --logError --logDebug --edge_max 5 --kmer_size 16 --index_mode gcsa-mem --include_primary 'azure:hgvm:hgvmeval-jobstore33' --path_name 'ref' --path_size 81189 '/home/cmarkello/debug_eval_input/BRCA1.vg' '/home/cmarkello/debug_eval_input/BRCA1/NA12877/NA12877.bam.fq' 'NA12877' '/home/cmarkello/debug_eval_output' 'azure:hgvm:hgvmdebugtest-input' 'azure:hgvm:hgvmdebugtest-output'`

Toil script `main(args)` function:
`

```
if len(args) == 2 and args[1] == "--test":
    # Run the tests
    return doctest.testmod(optionflags=doctest.NORMALIZE_WHITESPACE)

RealTimeLogger.start_master()
options = parse_args(args) # This holds the nicely-parsed options object

with Toil(options) as toil:
    if not toil.options.restart:

        # Upload local files to the remote IO Store
        inputGraphFileID = toil.importFile('file://'+options.vg_graph)
        inputReadsFileID = toil.importFile('file://'+options.sample_reads)
        uploadList = [[inputGraphFileID, options.vg_graph], [inputReadsFileID, options.sample_reads]]

        # Make a root job
        root_job = Job.wrapJobFn(run_upload, options, uploadList, cores=2, memory="5G", disk="5G")

        # Run the job and store the returned list of output files to download
        outputFileIDList = toil.start(root_job)
    else:
        outputFileIDList = toil.restart()

    # Download output files to the local machine that runs this script
    run_download(toil, options, outputFileIDList)

print("All jobs completed successfully")

RealTimeLogger.stop_master()`
```


Changed files:
src/toil/common.py

diff --git a/src/toil/common.py b/src/toil/common.py
index 911e86e3..caf6a4b3 100644
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -633,7 +633,7 @@ class Toil(object):
         elif name == 'azure':
             from toil.jobStores.azureJobStore import AzureJobStore
             account, namePrefix = rest.split(':', 1)
-            return AzureJobStore(account, namePrefix, config=config)
+            return AzureJobStore(rest)
 
         elif name == 'google':
             from toil.jobStores.googleJobStore import GoogleJobStore
Repository: databiosphere/toil
Issue: Nonetype object has no attribute "terminate"

An exception generated on interpreter shutdown while running SortTest. Seems benign but the exception should still be either avoided, caught, or documented. See http://stackoverflow.com/questions/1745232/solving-thread-cleanup-on-paramiko for more on this exception


Changed files:
src/toil/worker.py

diff --git a/src/toil/worker.py b/src/toil/worker.py
index 0c71cb3b..389282b1 100644
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -102,7 +102,14 @@ def main():
             os.kill(os.getpid(), signal.SIGKILL) #signal.SIGINT)
             #TODO: FIX OCCASIONAL DEADLOCK WITH SIGINT (tested on single machine)
         t = Thread(target=badWorker)
-        t.daemon = True
+        # Ideally this would be a daemon thread but that causes an intermittent (but benign)
+        # exception similar to the one described here:
+        # http://stackoverflow.com/questions/20596918/python-exception-in-thread-thread-1-most-likely-raised-during-interpreter-shutd
+        # Our exception is:
+        #    Exception in thread Thread-1 (most likely raised during interpreter shutdown):
+        #    <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'kill'
+        # This attribute error is caused by the call os.kill() and apparently unavoidable with a
+        # daemon
         t.start()
 
     ##########################################
Repository: databiosphere/toil
Issue: SSLError during SDB requests

http://jenkins.cgcloud.info/job/toil-pull-requests/1462/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

Needs retry.


Changed files:
src/toil/jobStores/aws/utils.py

diff --git a/src/toil/jobStores/aws/utils.py b/src/toil/jobStores/aws/utils.py
index cfbeabe3..e7186d80 100644
--- a/src/toil/jobStores/aws/utils.py
+++ b/src/toil/jobStores/aws/utils.py
@@ -20,6 +20,8 @@ import logging
 import types
 
 import errno
+from ssl import SSLError
+
 from boto.exception import (SDBResponseError,
                             BotoServerError,
                             S3ResponseError,
@@ -202,8 +204,16 @@ def no_such_sdb_domain(e):
             and e.error_code.endswith('NoSuchDomain'))
 
 
+def retryable_ssl_error(e):
+    # https://github.com/BD2KGenomics/toil/issues/978
+    return isinstance(e, SSLError) and e.reason == 'DECRYPTION_FAILED_OR_BAD_RECORD_MAC'
+
+
 def retryable_sdb_errors(e):
-    return sdb_unavailable(e) or no_such_sdb_domain(e) or connection_reset(e)
+    return (sdb_unavailable(e)
+            or no_such_sdb_domain(e)
+            or connection_reset(e)
+            or retryable_ssl_error(e))
 
 
 def retry_sdb(delays=default_delays, timeout=default_timeout, predicate=retryable_sdb_errors):
@@ -223,21 +233,21 @@ def retry_s3(delays=default_delays, timeout=default_timeout, predicate=retryable
     return retry(delays=delays, timeout=timeout, predicate=predicate)
 
 
-def region_to_bucket_location( region ):
+def region_to_bucket_location(region):
     if region == 'us-east-1':
         return ''
     else:
         return region
 
 
-def bucket_location_to_region( location ):
+def bucket_location_to_region(location):
     if location == '':
         return 'us-east-1'
     else:
         return location
 
 
-def bucket_location_to_http_url( location ):
+def bucket_location_to_http_url(location):
     if location:
         return 'https://s3-' + location + '.amazonaws.com'
     else:
Repository: databiosphere/toil
Issue: Minor copy-paste error in worker.py

I just stumbled across this:

https://github.com/BD2KGenomics/toil/blob/5ec9a2cd1fb081bd59a93a7aa0023d19cab6750b/src/toil/worker.py#L496

``` python
    #Close redirected stderr and replace with the original standard error.
    os.dup2(origStdOut, 2)
```

That should read `os.dup2(origStdErr, 2)` if you want to actually restore the original stderr to fd 2. I can't think of a way that this would ever cause a problem, though.


Changed files:
src/toil/worker.py

diff --git a/src/toil/worker.py b/src/toil/worker.py
index ddcfbfd5..9cc3f37b 100644
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -493,7 +493,7 @@ def main():
     os.dup2(origStdOut, 1)
     
     #Close redirected stderr and replace with the original standard error.
-    os.dup2(origStdOut, 2)
+    os.dup2(origStdErr, 2)
     
     #sys.stdout and sys.stderr don't need to be modified at all. We don't need
     #to call redirectLoggerStreamHandlers since they still log to sys.stderr
Repository: databiosphere/toil
Issue: Node preemptability is determined inconsistently

The batch system looks for a text value for preemptability while the cluster scaler code looks for a scalar value.

Changed files:
src/toil/batchSystems/mesos/batchSystem.py

diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
index 93c03468..38902e39 100644
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -437,7 +437,7 @@ class MesosBatchSystem(BatchSystemSupport,
             preemptable = False
             for attribute in offer.attributes:
                 if attribute.name == 'preemptable':
-                    preemptable = bool(attribute.scalar.value)
+                    preemptable = strict_bool(attribute.text.value)
             if preemptable:
                 try:
                     self.nonPreemptibleNodes.remove(offer.slave_id.value)
Repository: databiosphere/toil
Issue: `toil clean` doesn't clean up job stores on Azure

I've been running jobs with Azure job stores, and I've been wanting to restart them from scratch. I'm supposed to be able to do this with `toil clean`, but it doesn't work.

```
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ toil clean azure:hgvm:amntree
amn2master1 2016-08-29 20:12:58,971 MainThread INFO toil.lib.bioio: Logging set at level: INFO
amn2master1 2016-08-29 20:12:58,971 MainThread INFO toil.utils.toilClean: Attempting to delete the job store
amn2master1 2016-08-29 20:12:59,034 MainThread INFO toil.utils.toilClean: Successfully deleted the job store
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ scripts/parallelMappingEvaluation.py azure:hgvm:amntree graph_urls.tsv azure:hgvm:pilot-input/reads azure:hgvm:pilot-output/low-coverage/aug16/out1 --batchSystem=mesos --mesosMaster 10.0.0.5:5050 --kmer_size=16 --edge_max=3 --index_mode gcsa-mem --include_primary --sample_limit 10 --disableCaching --workDir /tmp 2>&1 | tee log.txt
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): hgvm.table.core.windows.net
Traceback (most recent call last):
  File "scripts/parallelMappingEvaluation.py", line 1170, in <module>
    sys.exit(main(sys.argv))
  File "scripts/parallelMappingEvaluation.py", line 1160, in main
    failed_jobs = Job.Runner.startToil(root_job,  options)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 540, in startToil
    with Toil(options) as toil:
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 501, in __enter__
    jobStore.initialize(config)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/azureJobStore.py", line 152, in initialize
    raise JobStoreExistsException(self.locator)
toil.jobStores.abstractJobStore.JobStoreExistsException: The job store 'hgvm:amntree' already exists. Use --restart to resume the workflow, or remove the job store with 'toil clean' to start the workflow from scratch
anovak@amn2master1:~/hgvm-graph-bakeoff-evaluations$ 
```

Thinking it could have been a timing thing with Azure, I've tried waiting a few minutes between the clean command and trying to re-use the job store name, but it hasn't helped. Clean is still reporting success, and Toil is still telling me that I need to run clean.


Changed files:
src/toil/jobStores/azureJobStore.py

diff --git a/src/toil/jobStores/azureJobStore.py b/src/toil/jobStores/azureJobStore.py
index b8d0c875..5a16194d 100644
--- a/src/toil/jobStores/azureJobStore.py
+++ b/src/toil/jobStores/azureJobStore.py
@@ -165,6 +165,7 @@ class AzureJobStore(AbstractJobStore):
         super(AzureJobStore, self).resume()
 
     def destroy(self):
+        self._bind()
         for name in 'jobItems', 'jobFileIDs', 'files', 'statsFiles', 'statsFileIDs':
             resource = getattr(self, name)
             if resource is not None:
Repository: databiosphere/toil
Issue: Log level must be explicitly set for RealtimeLogger to work

I have logging statements in my user code that log through the Toil RealtimeLogger. However, the RealtimeLogger doesn't get initialized unless I specify one of the Toil log level options (--logInfo, --logWarning, etc.). Without those I just get a message that it is disabled, even when I pass --realTimeLogging.

Instead, the real time logger should be enabled with the default log level if --realTimeLogging alone is passed.

Changed files:
src/toil/lib/bioio.py

diff --git a/src/toil/lib/bioio.py b/src/toil/lib/bioio.py
index e70ef878..0f6ba07a 100644
--- a/src/toil/lib/bioio.py
+++ b/src/toil/lib/bioio.py
@@ -124,12 +124,15 @@ def _addLoggingOptions(addOptionFn):
     # BEFORE YOU ADD OR REMOVE OPTIONS TO THIS FUNCTION, KNOW THAT YOU MAY ONLY USE VARIABLES ACCEPTED BY BOTH
     # optparse AND argparse FOR EXAMPLE, YOU MAY NOT USE default=%default OR default=%(default)s
     defaultLogLevelName = logging.getLevelName( defaultLogLevel )
-    addOptionFn("--logOff", dest="logCritical", action="store_true", default=False,
+    addOptionFn("--logOff", dest="logLevel",
+                default=defaultLogLevelName,
+                action="store_const", const="CRITICAL",
                 help="Same as --logCritical")
     for level in supportedLogLevels:
         levelName = logging.getLevelName(level)
         levelNameCapitalized = levelName.capitalize()
         addOptionFn("--log" + levelNameCapitalized, dest="logLevel",
+                    default=defaultLogLevelName,
                     action="store_const", const=levelName,
                     help="Turn on logging at level %s and above. (default is %s)" % (levelName, defaultLogLevelName))
     addOptionFn("--logLevel", dest="logLevel", default=defaultLogLevelName,
Repository: databiosphere/toil
Issue: nested directory getting lost between steps

hi! I'm unsuccessfully trying to run our CWL pipeline with toil. It looks like somehow intermediate date products get lost. The pipeline runs fine with the CWL reference runner. My logfile is filled with errors like:

```
2017-11-26 11:19:34,177 - toil.leader - WARNING - W/2/jobLmwkno    WorkflowException: [Errno 2] No such file or directory: u'/vault-tina/gijs/prefactor-cwl/runs/run_2017-11-26-11-18-25/results/out_tmpdir2RF79v/calibrated.MS/OBSERVATION/table.lock'
```

The intermediate folder seems to be empty. My guess this is probably related to our data products being nested directory structures and maybe in combination with docker.

To reproduce this issue you could:
* clone our repository https://github.com/EOSC-LOFAR/prefactor-cwl/tree/b5c080d10c659232b46dd9dcb2d9e4c19cc9febc
* run `make toil` in the checkout

Requirements:
 * git-lfs
 * Docker
 * virtualenv





Changed files:
src/toil/cwl/cwltoil.py

diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
index 554747cd..cf01e686 100755
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -455,12 +455,6 @@ class CWLJob(Job):
                                                 cwltool.stdfsaccess.StdFsAccess(outdir),
                                                 recursive=True))
 
-        def make_dir_literal(obj):
-            if "location" in obj and obj["location"].startswith("file:"):
-                obj["location"] = "_:" + str(uuid.uuid4())
-
-        adjustDirObjs(output, make_dir_literal)
-
         adjustFileObjs(output, functools.partial(uploadFile,
                                                  functools.partial(writeGlobalFileWrapper, fileStore),
                                                  index, existing))
Repository: databiosphere/toil
Issue: Error removing toil/src directory from python path

When installed from pip (pip install git+http://github.com/bd2kgenomics/toil) into a virtualenv:

> Added job: b/k/jobHAlTgv to active jobs
> Built the jobs list, currently have 1 jobs to update and 0 jobs issued
> Issuing the command: /home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv with memory: 2147483648, cores: >1, disk: 2147483648
> Issued job with job store ID: b/k/jobHAlTgv and job batch system ID: 98 and cores: 1, disk: >2147483648, and memory: 2147483648
> Acquiring 2147483648 bytes of memory from pool of 9223372036854775807.
> Attempting to acquire 10 threads for 1 cpus submitted
> Acquiring semaphore blockingly.
> Semaphore acquired.
> Executing command: '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.
> Traceback (most recent call last):
>  File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >24, in <module>
>    sys.path.remove(os.path.dirname(os.path.abspath(**file**)))
> ValueError: list.remove(x): x not in list
> Got exit code 1 (indicating failure) from command '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.


Changed files:
src/toil/worker.py

diff --git a/src/toil/worker.py b/src/toil/worker.py
index fe0bb1d4..13948bd9 100644
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -21,7 +21,8 @@ import sys
 if __name__ == "__main__":
     # FIXME: Until we use setuptools entry points, this is the only way to avoid a conflict between our own resource.py
     # and Python's
-    sys.path.remove(os.path.dirname(os.path.abspath(__file__)))
+    toilSrcDir = os.path.dirname(os.path.realpath(__file__))
+    sys.path = [directory for directory in sys.path if not os.path.realpath(directory) == toilSrcDir]
 
 import traceback
 import time
Repository: databiosphere/toil
Issue: Sort test has slowed down

Runtime doubled to 13min. It started with my first stab at #274 (#275) and so I thought it was caused by it. Since then I made the existence check faster and more reliable (#283) but that didn't seem to have an effect. 

I did notice some "SDB is not available errors" today so maybe we should wait a bit.


Changed files:
src/toil/jobStores/awsJobStore.py

diff --git a/src/toil/jobStores/awsJobStore.py b/src/toil/jobStores/awsJobStore.py
index b79589f9..a381e89a 100644
--- a/src/toil/jobStores/awsJobStore.py
+++ b/src/toil/jobStores/awsJobStore.py
@@ -34,7 +34,7 @@ from boto.s3.connection import S3Connection
 from boto.sdb.connection import SDBConnection
 from boto.sdb.item import Item
 import boto.s3
-from boto.exception import SDBResponseError, S3ResponseError
+from boto.exception import SDBResponseError, S3ResponseError, BotoServerError
 import itertools
 import time
 
@@ -245,7 +245,7 @@ class AWSJobStore(AbstractJobStore):
         jobStoreFileID = self._newFileID(sharedFileName)
         oldVersion = self._getFileVersion(jobStoreFileID)
         with self._uploadStream(jobStoreFileID, self.files,
-                                encrypted=isProtected) as ( writable, key):
+                                encrypted=isProtected) as (writable, key):
             yield writable
         newVersion = key.version_id
         jobStoreId = str(self.sharedFileJobID) if oldVersion is None else None
@@ -424,7 +424,15 @@ class AWSJobStore(AbstractJobStore):
 
         :rtype : Domain
         """
-        return self.db.create_domain(domain_name)
+        try:
+            return self.db.get_domain(domain_name)
+        except SDBResponseError as e:
+            if no_such_domain(e):
+                for attempt in retry_sdb(retry_while=sdb_unavailable):
+                    with attempt:
+                        return self.db.create_domain(domain_name)
+            else:
+                raise
 
     def _newJobID(self):
         return str(uuid.uuid4())
@@ -724,7 +732,11 @@ a_long_time = 60 * 60
 
 
 def no_such_domain(e):
-    return e.error_code.endswith('NoSuchDomain')
+    return isinstance(e, SDBResponseError) and e.error_code.endswith('NoSuchDomain')
+
+
+def sdb_unavailable(e):
+    return e.__class__ == BotoServerError and e.status.startswith("503")
 
 
 def true(_):
@@ -806,7 +818,7 @@ def retry_sdb(retry_after=a_short_time,
         def repeated_attempt():
             try:
                 yield
-            except SDBResponseError as e:
+            except BotoServerError as e:
                 if time.time() + retry_after < expiration:
                     if retry_while(e):
                         log.info('... got %s, trying again in %is ...', e.error_code, retry_after)
Repository: databiosphere/toil
Issue: User script hot deployment not functioning

WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    ---JOBTREE WORKER OUTPUT LOG---
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    Traceback (most recent call last):
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/worker.py", line 267, in main
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        jobStore=jobStore)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/target.py", line 714, in _execute
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        returnValues = self.run(fileStore)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/target.py", line 818, in run
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        userFunction = self._getUserFunction()
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/target.py", line 791, in _getUserFunction
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        return getattr(importlib.import_module(userFunctionModule.name), self.userFunctionName)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        **import**(name)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    ImportError: No module named testUserScript
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    Exiting the worker because of a failed job on host ip-172-31-21-198
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    ERROR:**main**:Exiting the worker because of a failed job on host ip-172-31-21-198
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    WARNING:jobTree.job:Due to failure we are reducing the remaining retry count of job 2fa1bc1d-7dec-4b88-814f-42bfcc159a2b to 0


Changed files:
src/toil/job.py

diff --git a/src/toil/job.py b/src/toil/job.py
index 574f3550..ebe77fe5 100644
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -342,7 +342,7 @@ class Job(object):
             Raises an exception if the given toil already exists.
             """
             setLoggingFromOptions(options)
-            with setupToil(options) as (config, batchSystem, jobStore):
+            with setupToil(options, userScript=job.getUserScript()) as (config, batchSystem, jobStore):
                 jobStore.clean()
                 if "rootJob" not in config.attrib: #No jobs have yet been run
                     # Setup the first batchjob.
Repository: databiosphere/toil
Issue: resolveEntryPoint() fails on a system-wide installation of Toil

Python is installed in `/usr/bin` while pip installed the entry points in `/usr/local/bin`. That's why the lookup logic does not work.


Changed files:
src/toil/__init__.py

diff --git a/src/toil/__init__.py b/src/toil/__init__.py
index 8aafed57..4616d744 100644
--- a/src/toil/__init__.py
+++ b/src/toil/__init__.py
@@ -27,6 +27,23 @@ def toilPackageDirPath():
     assert result.endswith('/toil')
     return result
 
-def resolveEntryPoint(entryPoint):
-    return os.path.join(os.path.dirname(sys.executable),entryPoint)
 
+def resolveEntryPoint(entryPoint):
+    """
+    Returns the path to the given entry point (see setup.py) that *should* work on a worker. The
+    return value may be an absolute or a relative path.
+    """
+    if hasattr(sys, 'real_prefix'):
+        # Inside a virtualenv we will use absolute paths to the entrypoints. For clusters this
+        # means that if Toil is installed in a virtualenv on the leader, it must be installed in
+        # a virtualenv located at the same path on the worker.
+        path = os.path.join(os.path.dirname(sys.executable), entryPoint)
+        assert os.path.isfile(path)
+        assert os.access(path, os.X_OK)
+        return path
+    else:
+        # Outside a virtualenv it is hard to predict where the entry points got installed. It is
+        # the reponsibility of the user to ensure that they are present on PATH and point to the
+        # correct version of Toil. This is still better than an absolute path because it gives
+        # the user control over Toil's location on both leader and workers.
+        return entryPoint
Repository: databiosphere/toil
Issue: Mesos batch system doesn't favor bigger tasks

https://github.com/BD2KGenomics/toil/blob/d08fe8c43d0d18d4d4bdafb9c9c921fd937557bd/src/toil/batchSystems/mesos/batchSystem.py#L298

Sort is ineffective as it sorts by the property descriptor, not the value of the property.


Changed files:
src/toil/batchSystems/mesos/batchSystem.py

diff --git a/src/toil/batchSystems/mesos/batchSystem.py b/src/toil/batchSystems/mesos/batchSystem.py
index 198f14db..95d565d7 100644
--- a/src/toil/batchSystems/mesos/batchSystem.py
+++ b/src/toil/batchSystems/mesos/batchSystem.py
@@ -295,7 +295,7 @@ class MesosBatchSystem(AbstractBatchSystem, mesos.interface.Scheduler):
         job_types = self.jobQueueList.keys()
         # sorts from largest to smallest core usage
         # TODO: add a size() method to ResourceSummary and use it as the key. Ask me why.
-        job_types.sort(key=lambda resourceRequirement: ResourceRequirement.cores)
+        job_types.sort(key=lambda resourceRequirement: resourceRequirement.cores)
         job_types.reverse()
         return job_types
Repository: databiosphere/toil
Issue: cwltoil raises schema-related ValidationException

Tests started failing just now.

http://jenkins.cgcloud.info/job/toil-pull-requests/lastCompletedBuild/testReport/src.toil.test.cwl.cwlTest/CWLTest/test_run_revsort/

It would be nice if the schema didn't change underneath a particular cwl version. IOW, if I pin my cwl version, the schema used by it should never change. But that's just my hunch. I don't understand the intricacies involved here.

@tetron would you have time to take a look?


Changed files:
setup.py, src/toil/cwl/cwltoil.py, src/toil/test/cwl/cwlTest.py

diff --git a/setup.py b/setup.py
index 386fe851..edf5d0b3 100644
--- a/setup.py
+++ b/setup.py
@@ -39,7 +39,7 @@ kwargs = dict(
         'encryption': [
             'pynacl==0.3.0'],
         'cwl': [
-            'cwltool==1.0.20151122025918']},
+            'cwltool==1.0.20151211141743']},
     package_dir={'': 'src'},
     packages=find_packages('src', exclude=['*.test']),
     entry_points={
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
index f68bb99c..3eac1537 100755
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -412,7 +412,8 @@ def main(args=None):
 
     job, _ = loader.resolve_ref(uri)
 
-    t = cwltool.main.load_tool(options.cwltool, False, False, cwltool.workflow.defaultMakeTool,
+    t = cwltool.main.load_tool(options.cwltool, False, True,
+                               cwltool.workflow.defaultMakeTool,
                                True)
 
     if type(t) == int:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
index 26925a43..05e524ec 100644
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -20,7 +20,6 @@ from toil.test import ToilTest, needs_cwl
 
 @needs_cwl
 class CWLTest(ToilTest):
-    @skip("https://github.com/BD2KGenomics/toil/issues/621")
     def test_run_revsort(self):
         from toil.cwl import cwltoil
         outDir = self._createTempDir()
Repository: robocorp/rpaframework
Issue: `Add to JSON` keyword in `RPA.JSON` library fails to add values into JSON structure

Failing example 1.

```robot
*** Tasks ***
    ${json}   Convert String to JSON  {}
    ${json}   Add to JSON   ${json}  $.FOOBAR  Hello World
    ${text}   Convert To String  ${json}
    notebook print  ${text}
```

Failing example 2. (from keyword documentation)

```robot
*** Tasks ***
Change the name value for all people
   &{before}=    Convert string to JSON   {"People": [{"Name": "Mark"}, {"Name": "Jane"}]}
   &{after}=     Add to JSON    ${before}   $.People.Name    JohnMalkovich
    notebook print  ${text}
```

In addition the library documentation (examples) need to be fixed.

Changed files:
packages/main/src/RPA/JSON.py

diff --git a/packages/main/src/RPA/JSON.py b/packages/main/src/RPA/JSON.py
index 6baef40bf..8923bc393 100644
--- a/packages/main/src/RPA/JSON.py
+++ b/packages/main/src/RPA/JSON.py
@@ -233,9 +233,10 @@ class JSON:
 
         .. code:: robotframework
 
-           # Change the name value for all people
+           # Add new person to list
+           &{person}=    Create dictionary    name=John
            &{before}=    Convert string to JSON   {"People": [{"Name": "Mark"}, {"Name": "Jane"}]}
-           &{after}=     Add to JSON    ${json}   $.People.name    JohnMalkovich
+           &{after}=     Add to JSON    ${before}   $.People    ${person}
 
         """  # noqa: E501
         self.logger.info('Add to JSON with expression: "%s"', expr)
@@ -263,7 +264,7 @@ class JSON:
 
            # Get the name value for the first person
            &{people}=    Convert string to JSON   {"People": [{"Name": "Mark"}, {"Name": "Jane"}]}
-           ${first}=     Get value from JSON      ${people}   $.People[0].name
+           ${first}=     Get value from JSON      ${people}   $.People[0].Name
 
         """  # noqa: E501
         self.logger.info('Get value from JSON with expression: "%s"', expr)
@@ -292,7 +293,7 @@ class JSON:
 
            # Get all the names for all people
            &{people}=    Convert string to JSON   {"People": [{"Name": "Mark"}, {"Name": "Jane"}]}
-           @{names}=     Get values from JSON     ${people}   $.People[*].name
+           @{names}=     Get values from JSON     ${people}   $.People[*].Name
 
         """  # noqa: E501
         self.logger.info('Get values from JSON with expression: "%s"', expr)
@@ -313,7 +314,7 @@ class JSON:
 
            # Change the name key for all people
            &{before}=    Convert string to JSON   {"People": [{"Name": "Mark"}, {"Name": "Jane"}]}
-           &{after}=     Update value to JSON     ${json}   $.People[*].name    JohnMalkovich
+           &{after}=     Update value to JSON     ${before}   $.People[*].Name    JohnMalkovich
 
         """  # noqa: E501
         self.logger.info('Update JSON with expression: "%s"', expr)