Repository: DataBiosphere/toil

Issue:
Unable to run toil launch-cluster with version 5.2
I am attempting to finally migrate cactus from toil 4.2 to toil 5.2, and am running into problems.  First, using my usual command to make a cluster, I get this Invalid provisioner error:

```
toil launch-cluster glenn-toil52 --leaderNodeType t2.medium --keyPairName glenn.hickey@gmail.com
Traceback (most recent call last):
  File "/home/hickey/dev/cactus/venv52/bin/toil", line 8, in <module>
    sys.exit(main())
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/utils/toilMain.py", line 26, in main
    module.main()
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/utils/toilLaunchCluster.py", line 99, in main
    check_valid_node_types(options.provisioner, worker_node_types + [options.leaderNodeType])
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/provisioners/__init__.py", line 102, in check_valid_node_types
    raise RuntimeError(f"Invalid provisioner: {provisioner}")
RuntimeError: Invalid provisioner: None
```

Despite being a new requirement, this option is not described in the short help
```
usage: Toil [-h] [-p {aws,gce}] [-z ZONE] [--logCritical] [--logError]
            [--logWarning] [--logDebug] [--logInfo] [--logOff]
            [--logLevel {Critical,Error,Warning,Debug,Info,critical,error,warning,debug,info,CRITICAL,ERROR,WARNING,DEBUG,INFO}]
            [--logFile LOGFILE] [--rotatingLogging] [--version]
            [--tempDirRoot TEMPDIRROOT] --leaderNodeType LEADERNODETYPE
            [--keyPairName KEYPAIRNAME] [--owner OWNER] [--boto BOTOPATH]
            [-t NAME=VALUE] [--vpcSubnet VPCSUBNET] [--nodeTypes NODETYPES]
            [-w WORKERS] [--leaderStorage LEADERSTORAGE]
            [--nodeStorage NODESTORAGE] [--forceDockerAppliance]
            [--awsEc2ProfileArn AWSEC2PROFILEARN]
            [--awsEc2ExtraSecurityGroupId AWSEC2EXTRASECURITYGROUPIDS]
            clusterName
```
and while I'm griping, perhaps a more explicit error for failing to provide a required option would be helpful? 

When I try running with `--provisioner aws`  I get this
```
toil launch-cluster glenn-toil52 --leaderNodeType t2.medium --keyPairName glenn.hickey@gmail.com --provisioner aws
[2021-01-08T16:12:21-0500] [MainThread] [I] [toil] Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
[2021-01-08T16:12:21-0500] [MainThread] [I] [toil] Using default docker name of toil as TOIL_DOCKER_NAME is not set.
[2021-01-08T16:12:21-0500] [MainThread] [I] [toil] Using default docker appliance of quay.io/ucsc_cgl/toil:5.2.0-047d0c4f2949c576c80e452a0807c5be6355c63d-py3.6 as TOIL_APPLIANCE_SELF is not set.
[2021-01-08T16:12:26-0500] [MainThread] [I] [toil] Using default user-defined custom docker init command of  as TOIL_CUSTOM_DOCKER_INIT_COMMAND is not set.
[2021-01-08T16:12:26-0500] [MainThread] [I] [toil] Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
[2021-01-08T16:12:26-0500] [MainThread] [I] [toil] Using default docker name of toil as TOIL_DOCKER_NAME is not set.
[2021-01-08T16:12:26-0500] [MainThread] [I] [toil] Using default docker appliance of quay.io/ucsc_cgl/toil:5.2.0-047d0c4f2949c576c80e452a0807c5be6355c63d-py3.6 as TOIL_APPLIANCE_SELF is not set.
[2021-01-08T16:12:26-0500] [MainThread] [I] [toil] Using default user-defined custom init command of  as TOIL_CUSTOM_INIT_COMMAND is not set.
[2021-01-08T16:12:26-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ...
```
Then it asks me for my MFA code, I paste it in, then it asks me again (I've tried several times)
```
Enter MFA code for arn:aws:iam::652235167018:mfa/glenn.hickey@gmail.com: 
Enter MFA code for arn:aws:iam::652235167018:mfa/glenn.hickey@gmail.com:
```
Then it eventually fails presumably because it didn't like my MFA code? 
```
[2021-01-08T16:12:43-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
[2021-01-08T16:12:48-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
[2021-01-08T16:12:58-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
[2021-01-08T16:13:18-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
[2021-01-08T16:13:38-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
[2021-01-08T16:13:58-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
[2021-01-08T16:14:19-0500] [MainThread] [I] [toil.lib.ec2] Creating t2.medium instance(s) ... 
Traceback (most recent call last):
  File "/home/hickey/dev/cactus/venv52/bin/toil", line 8, in <module>
    sys.exit(main())
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/utils/toilMain.py", line 26, in main
    module.main()
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/utils/toilLaunchCluster.py", line 132, in main
    awsEc2ExtraSecurityGroupIds=options.awsEc2ExtraSecurityGroupIds)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/provisioners/aws/awsProvisioner.py", line 200, in launchCluster
    subnet_id=self._vpcSubnet)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/lib/retry.py", line 246, in call
    return func(*args, **kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/toil/lib/ec2.py", line 313, in create_instances
    return ec2.create_instances(**actual_request)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/signers.py", line 154, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/signers.py", line 234, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/credentials.py", line 612, in get_frozen_credentials
    self._refresh()
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/credentials.py", line 507, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/credentials.py", line 523, in _protected_refresh
    metadata = self._refresh_using()
  File "/home/hickey/dev/cactus/venv52/lib/python3.6/site-packages/botocore/credentials.py", line 269, in __call__
    raise RefreshWithMFAUnsupportedError()
botocore.exceptions.RefreshWithMFAUnsupportedError: Cannot refresh credentials: MFA token required.
```
I have never seen anything like this with toil 4.2.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-776)
┆Issue Number: TOIL-776



Discussion:
@glennhickey Will make a fix.
@glennhickey This will be fixed in https://github.com/DataBiosphere/toil/pull/3357 .
@glennhickey MFA should be solved.  Could you test it?
Just tried master and it worked, thanks.  I ran the cactus example with the autoscaler and it also worked, but got failed-with-exit-0 warning  like this for pretty much every job
```
[2021-02-03T18:10:15+0000] [MainThread] [W] [toil.leader] Job failed with exit value 0: 'JobFunctionWrappingJob' 9c385232-a7b5-4a69-92c7-f62c98534778
```
@glennhickey That's odd.  It'd be nice to have a stack trace or worker logs.  I suppose fixing your error is probably going to be two tickets, exposing the error immediately to the user without having to dig, and actually solving whatever is happening.

A stack trace would be helpful, but lacking that, if you post your run steps for me to reproduce, I can try to debug this next week.
It never crashed, so I don't have a stack trace.  Steps are pretty much the same as described here https://github.com/DataBiosphere/toil/issues/2968#issue-563423341  except installing toil from "git+https://github.com/DataBiosphere/toil" and running on the cactus master branch.  
@glennhickey Thanks!
@glennhickey I've reproduced running the workflow with the following:

```
git clone https://github.com/DataBiosphere/toil.git && cd toil && virtualenv -p python3.6 v3nv && . v3nv/bin/activate && make prepare && make develop extras=[all]
make push_docker
toil launch-cluster lontestcactor -p aws --leaderNodeType=t2.medium -z us-west-2a --keyPairName=id_rsa
toil ssh-cluster -p aws -z us-west-2a lontestcactor
```

```
screen
mkdir /tmp/test
cd /tmp/test
virtualenv -p python3.6 --system-site-packages venv
. venv/bin/activate
git clone https://github.com/ComparativeGenomicsToolkit/cactus.git --recursive
cd cactus
pip install -U .
cactus  --nodeTypes c4.8xlarge:0.6,r3.8xlarge --minNodes 0,0 --maxNodes 20,2 --provisioner aws --batchSystem mesos --logDebug --latest aws:us-west-2:loncactustestdelete ./examples/evolverMammals.txt output.hal  --logFile log --clean never &> loncactus.feb09.log
```

This seems(?) successful and creates an output.hal file.  I also get the weird warning message.

It looks like this may have uncovered unintended behavior, since this seems to have been changed by the CWL folks recently in a way that allowed the warning to print: https://github.com/DataBiosphere/toil/blame/98dbf33147ed029800dd43a73ee5c64e83feda7b/src/toil/leader.py#L568

It's not printing the exitReason, which might make this clearer, but looking at it a little deeper, this doesn't seem to be an error that cactus needs to worry about.  It seems like multiple things trying to update a job and failing because it's already finished, which would just mean we're being inefficient.

I'll dig a little deeper into this tomorrow and try to see if I can make this a bit better.

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -410,10 +410,13 @@ def addOptions(parser: ArgumentParser, config: Config = Config()):
         description="Allows the specification of the minimum and maximum number of nodes in an autoscaled cluster, "
                     "as well as parameters to control the level of provisioning."
     )
-    provisioner_choices = ['aws', 'gce']
+    provisioner_choices = ['aws', 'gce', None]
+    # TODO: Better consolidate this provisioner arg and the one in provisioners/__init__.py?
     autoscaling_options.add_argument('--provisioner', '-p', dest="provisioner", choices=provisioner_choices,
-                                     help=f"The provisioner for cluster auto-scaling. The currently supported choices "
-                                          f"are {provisioner_choices}.  The default is {config.provisioner}.")
+                                     help=f"The provisioner for cluster auto-scaling.  This is the main Toil "
+                                          f"'--provisioner' option, and defaults to None for running on single "
+                                          f"machine and non-auto-scaling batch systems.  The currently supported "
+                                          f"choices are {provisioner_choices}.  The default is {config.provisioner}.")
     autoscaling_options.add_argument('--nodeTypes', default=None,
                                      help="List of worker node types separated by commas. The syntax for each node "
                                           "type depends on the provisioner used.  For the AWS provisioner this is the "
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -564,12 +564,11 @@ def _gatherUpdatedJobs(self, updatedJobTuple):
         except KeyError:
             logger.warning("A result seems to already have been processed for job %s", jobID)
         else:
-            if exitStatus == 0 and exitReason == None:
-                cur_logger = (logger.debug if str(updatedJob.jobName).startswith(CWL_INTERNAL_JOBS)
-                              else logger.info)
-                cur_logger('Job ended: %s', updatedJob)
+            if exitStatus == 0:
+                logger.debug('Job ended: %s', updatedJob)
             else:
-                logger.warning(f'Job failed with exit value {exitStatus}: {updatedJob}')
+                logger.warning(f'Job failed with exit value {exitStatus}: {updatedJob}\n'
+                               f'Exit reason: {exitReason}')
             if self.toilMetrics:
                 self.toilMetrics.logCompletedJob(updatedJob)
             self.processFinishedJob(jobID, exitStatus, wallTime=wallTime, exitReason=exitReason)
diff --git a/src/toil/provisioners/__init__.py b/src/toil/provisioners/__init__.py
--- a/src/toil/provisioners/__init__.py
+++ b/src/toil/provisioners/__init__.py
@@ -52,10 +52,12 @@ def cluster_factory(provisioner, clusterName=None, clusterType='mesos', zone=Non
 def add_provisioner_options(parser):
     group = parser.add_argument_group("Provisioner Options.")
 
-    # TODO: Duplicate "--provisioner" argument in common.py; consolidate
-    group.add_argument('-p', '--provisioner', dest='provisioner', choices=['aws', 'gce'],
-                help=f"The provisioner for cluster auto-scaling. The currently supported choices are"
-                     f"'gce', or 'aws'.")
+    provisioner_choices = ['aws', 'gce']
+    # TODO: Better consolidate this provisioner arg and the one in common.py?
+    group.add_argument('--provisioner', '-p', dest="provisioner", choices=provisioner_choices, default='aws',
+                       help=f"The provisioner for cluster auto-scaling.  This is the '--provisioner' option set for "
+                            f"Toil utils like launch-cluster and destroy-cluster, which always require a provisioner, "
+                            f"and so this defaults to: %(default)s.  Choices: {provisioner_choices}.")
     group.add_argument('-z', '--zone', dest='zone', required=False, default=None,
                        help="The availability zone of the master. This parameter can also be set via the 'TOIL_X_ZONE' "
                             "environment variable, where X is AWS or GCE, or by the ec2_region_name parameter "


Test patch: