Repository: DataBiosphere/toil

Issue:
Toil gets in infinite loop trying to connect to server if the user's ssh ID has a passphrase on it
I had a passphrase on my `~/.ssh/id_rsa`, and Toil would get into a loop repeating this forever until I removed the passphrase:

```
2018-01-05 16:26:52,265 - toil.provisioners.aws.awsProvisioner - INFO - Attempting to establish SSH connection...
Warning: Permanently added 'ec2-35-167-98-105.us-west-2.compute.amazonaws.com,35.167.98.105' (ECDSA) to the list of known hosts.
Permission denied (publickey,password,keyboard-interactive).
2018-01-05 16:26:52,853 - toil.provisioners.aws.awsProvisioner - INFO - Connection rejected, waiting for public SSH key to be propagated. Trying again in 10s.
```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-223)
┆Issue Number: TOIL-223



Discussion:
Toil blocks on any runtime error until it connects, so this is a known issue.  It could definitely be handled better with smarter error handling.  I'll try to fix this for the passphrase case.
@cmarkello just ran into this too; if it can't be fixed it at least needs to be documented in the hello world tutorial.
So @adamnovak gave me a workaround which works, but is probably not ideal security-wise. What worked was resetting the password to decrypting the private ssh key to null via `ssh-keygen -p`.
Hi.  I followed the instructions at:  https://toil.readthedocs.io/en/latest/running/cloud/amazon.html, and I am getting the same error, e.g. 

INFO:toil.provisioners.node:Connection rejected, waiting for public SSH key to be propagated. Trying again in 10s.

I tried removing my passphrase, and uploaded the new public key to AWS, but am still getting the same error message.

Any more suggestions?
What do you mean by "uploaded the new public key to AWS"? If you remove the passphrase on the private key, there shouldn't be a new public key; you would keep the same public key as before.

When Toil is trying to ssh into the instance, can you get the IP of the instance from the AWS console and SSH into it manually? If not, there's something wrong with your SSH key configuration still. What's in your `~/.ssh` directory?

Resolving patch:
diff --git a/src/toil/provisioners/node.py b/src/toil/provisioners/node.py
--- a/src/toil/provisioners/node.py
+++ b/src/toil/provisioners/node.py
@@ -120,14 +120,21 @@ def extractFile(self, fromFile, toFile, role):
     def _waitForSSHKeys(self, keyName='core'):
         # the propagation of public ssh keys vs. opening the SSH port is racey, so this method blocks until
         # the keys are propagated and the instance can be SSH into
-        startTime = time.time()
+        start_time = time.time()
+        last_error = None
         while True:
-            if time.time() - startTime > self.maxWaitTime:
-                raise RuntimeError("Key propagation failed on machine with ip %s" % self.effectiveIP)
+            if time.time() - start_time > self.maxWaitTime:
+                raise RuntimeError(f"Key propagation failed on machine with ip {self.effectiveIP}." +
+                                   ("\n\nMake sure that your public key is attached to your account and you are using "
+                                    "the correct private key. If you are using a key with a passphrase, be sure to "
+                                    "set up ssh-agent. For details, refer to "
+                                    "https://toil.readthedocs.io/en/latest/running/cloud/cloud.html."
+                                    if last_error and 'Permission denied' in last_error else ""))
             try:
                 logger.info('Attempting to establish SSH connection...')
                 self.sshInstance('ps', sshOptions=['-oBatchMode=yes'], user=keyName)
-            except RuntimeError:
+            except RuntimeError as err:
+                last_error = str(err)
                 logger.info('Connection rejected, waiting for public SSH key to be propagated. Trying again in 10s.')
                 time.sleep(10)
             else:
@@ -240,8 +247,8 @@ def coreSSH(self, *args, **kwargs):
         # 3000 for Grafana dashboard
         # 9090 for Prometheus dashboard
         # 5050 for Mesos dashboard (although to talk to agents you will need a proxy)
-        commandTokens.extend(['-L', '3000:localhost:3000', \
-                              '-L', '9090:localhost:9090', \
+        commandTokens.extend(['-L', '3000:localhost:3000',
+                              '-L', '9090:localhost:9090',
                               '-L', '5050:localhost:5050'])
         if sshOptions:
             # add specified options to ssh command


Test patch: