Repository: DataBiosphere/toil

Issue:
realtimeLogger won't start up on my mac in single machine mode
The real time logger is upset that my FQDN is in the `in-addr.arpa` domain, and that it can't turn around and resolve it back to an IP.

```
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py:540: CalledProcessError
----------------------------- Captured stderr call -----------------------------
Vyoma 2017-09-14 10:27:56,718 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
Vyoma 2017-09-14 10:27:56,731 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '5ea1f538-be65-44d2-a615-4ba0d2cd52ee'
Vyoma 2017-09-14 10:27:58,959 MainThread INFO toil_vg.vg_index: Imported input files into Toil in 2.22710990906 seconds
Vyoma 2017-09-14 10:27:58,982 MainThread INFO toil.common: Using the single machine batch system
Vyoma 2017-09-14 10:27:58,983 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (4).
Vyoma 2017-09-14 10:27:58,983 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (8589934592).
Vyoma 2017-09-14 10:27:58,993 MainThread INFO toil.common: Created the workflow directory at /var/folders/ws/t7b9wm2j2b7fy4_xyq0n5vh00000gn/T/toil-5ea1f538-be65-44d2-a615-4ba0d2cd52ee-106618767760070
Vyoma 2017-09-14 10:27:58,993 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (16062173184).
Vyoma 2017-09-14 10:27:59,000 MainThread INFO toil.common: No user script to hot-deploy.
Vyoma 2017-09-14 10:27:59,001 MainThread INFO toil.common: Written the environment for the jobs to the environment file
Vyoma 2017-09-14 10:27:59,001 MainThread INFO toil.common: Caching all jobs in job store
Vyoma 2017-09-14 10:27:59,001 MainThread INFO toil.common: 0 jobs downloaded.
Vyoma 2017-09-14 10:27:59,009 MainThread INFO toil: Running Toil version 3.10.1-6fc52a6cce9103cfb38d150ce0fce73c610bc461.
Vyoma 2017-09-14 10:27:59,009 MainThread INFO toil.realtimeLogger: Starting real-time logging.
Traceback (most recent call last):
  File "/Users/anovak/Documents/Workspace/vg/.env/bin/toil-vg", line 9, in <module>
    load_entry_point('toil-vg==1.2.1a1', 'console_scripts', 'toil-vg')()
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil_vg/vg_toil.py", line 326, in main
    index_main(context, args)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil_vg/vg_index.py", line 469, in index_main
    index_key_and_id = toil.start(init_job)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/common.py", line 716, in start
    return self._runMainLoop(rootJobGraph)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/common.py", line 978, in _runMainLoop
    level=self.options.logLevel if self.options.realTimeLogging else None):
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/realtimeLogger.py", line 242, in __enter__
    RealtimeLogger._startLeader(self.__batchSystem, level=self.__level)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/realtimeLogger.py", line 148, in _startLeader
    ip = socket.gethostbyname(fqdn)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
=========================== 1 failed in 8.11 seconds ===========================
Running locally
Uploaded HTML as file://report-html/index.html
Markdown to post:
Jenkins vg tests complete.  View the [full report here](file://report-html/index.html).

0 tests passed, 1 tests failed and 0 tests skipped in 8 seconds

Failed tests:

* test_sim_brca1_snp1kg (6 seconds)
Vyoma:vg anovak$ python -i
Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.fqdn()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'fqdn'
>>> socket.getfqdn()
'1.0.0.127.in-addr.arpa'
>>> socket.gethostbyname(socket.getfqdn())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
>>> 
```

Toil may need special-case code to detect this vaguely fake FQDN and/or work around not being able to resolve it.
realtimeLogger won't start up on my mac in single machine mode
The real time logger is upset that my FQDN is in the `in-addr.arpa` domain, and that it can't turn around and resolve it back to an IP.

```
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py:540: CalledProcessError
----------------------------- Captured stderr call -----------------------------
Vyoma 2017-09-14 10:27:56,718 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
Vyoma 2017-09-14 10:27:56,731 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '5ea1f538-be65-44d2-a615-4ba0d2cd52ee'
Vyoma 2017-09-14 10:27:58,959 MainThread INFO toil_vg.vg_index: Imported input files into Toil in 2.22710990906 seconds
Vyoma 2017-09-14 10:27:58,982 MainThread INFO toil.common: Using the single machine batch system
Vyoma 2017-09-14 10:27:58,983 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (4).
Vyoma 2017-09-14 10:27:58,983 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (8589934592).
Vyoma 2017-09-14 10:27:58,993 MainThread INFO toil.common: Created the workflow directory at /var/folders/ws/t7b9wm2j2b7fy4_xyq0n5vh00000gn/T/toil-5ea1f538-be65-44d2-a615-4ba0d2cd52ee-106618767760070
Vyoma 2017-09-14 10:27:58,993 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (16062173184).
Vyoma 2017-09-14 10:27:59,000 MainThread INFO toil.common: No user script to hot-deploy.
Vyoma 2017-09-14 10:27:59,001 MainThread INFO toil.common: Written the environment for the jobs to the environment file
Vyoma 2017-09-14 10:27:59,001 MainThread INFO toil.common: Caching all jobs in job store
Vyoma 2017-09-14 10:27:59,001 MainThread INFO toil.common: 0 jobs downloaded.
Vyoma 2017-09-14 10:27:59,009 MainThread INFO toil: Running Toil version 3.10.1-6fc52a6cce9103cfb38d150ce0fce73c610bc461.
Vyoma 2017-09-14 10:27:59,009 MainThread INFO toil.realtimeLogger: Starting real-time logging.
Traceback (most recent call last):
  File "/Users/anovak/Documents/Workspace/vg/.env/bin/toil-vg", line 9, in <module>
    load_entry_point('toil-vg==1.2.1a1', 'console_scripts', 'toil-vg')()
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil_vg/vg_toil.py", line 326, in main
    index_main(context, args)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil_vg/vg_index.py", line 469, in index_main
    index_key_and_id = toil.start(init_job)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/common.py", line 716, in start
    return self._runMainLoop(rootJobGraph)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/common.py", line 978, in _runMainLoop
    level=self.options.logLevel if self.options.realTimeLogging else None):
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/realtimeLogger.py", line 242, in __enter__
    RealtimeLogger._startLeader(self.__batchSystem, level=self.__level)
  File "/Users/anovak/Documents/Workspace/vg/.env/lib/python2.7/site-packages/toil/realtimeLogger.py", line 148, in _startLeader
    ip = socket.gethostbyname(fqdn)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
=========================== 1 failed in 8.11 seconds ===========================
Running locally
Uploaded HTML as file://report-html/index.html
Markdown to post:
Jenkins vg tests complete.  View the [full report here](file://report-html/index.html).

0 tests passed, 1 tests failed and 0 tests skipped in 8 seconds

Failed tests:

* test_sim_brca1_snp1kg (6 seconds)
Vyoma:vg anovak$ python -i
Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.fqdn()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'fqdn'
>>> socket.getfqdn()
'1.0.0.127.in-addr.arpa'
>>> socket.gethostbyname(socket.getfqdn())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
>>> 
```

Toil may need special-case code to detect this vaguely fake FQDN and/or work around not being able to resolve it.


Discussion:
Everyone's been getting this for about a week now, tracked in #1858. Are you on macOS?

Given how many people are using master branch, we should really just revert or commit a band-aid fix.
Yeah, this is on OS X. I fixed it by adding an entry to hosts for that
nonsense DNS name.

What's weird is that it used to work, with my actual (non-fqdn) hostname in
/etc/hosts. Maybe I just didn't have the offending commit yet.

On Thu, Sep 14, 2017 at 12:02 PM, Joel Armstrong <notifications@github.com>
wrote:

> Everyone's been getting this for about a week now, tracked in #1858
> <https://github.com/BD2KGenomics/toil/issues/1858>. Are you on macOS?
>
> Given how many people are using master branch, we should really just
> revert or commit a band-aid fix.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1877#issuecomment-329356137>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AE0_X4hqHucsROJV4-QMBZzWvIhV3kwGks5siJdXgaJpZM4PW8Ba>
> .
>

@joelarmstrong revert which PR?
#1731, but actually I think I can get a proper fix in during the upcoming meeting. It's probably better to fix it properly than revert.
Actually, just noticed this has a slightly different stacktrace than #1858, though it has exactly the same cause, so reverting that PR wouldn't fix it anyway.
Everyone's been getting this for about a week now, tracked in #1858. Are you on macOS?

Given how many people are using master branch, we should really just revert or commit a band-aid fix.
Yeah, this is on OS X. I fixed it by adding an entry to hosts for that
nonsense DNS name.

What's weird is that it used to work, with my actual (non-fqdn) hostname in
/etc/hosts. Maybe I just didn't have the offending commit yet.

On Thu, Sep 14, 2017 at 12:02 PM, Joel Armstrong <notifications@github.com>
wrote:

> Everyone's been getting this for about a week now, tracked in #1858
> <https://github.com/BD2KGenomics/toil/issues/1858>. Are you on macOS?
>
> Given how many people are using master branch, we should really just
> revert or commit a band-aid fix.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1877#issuecomment-329356137>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AE0_X4hqHucsROJV4-QMBZzWvIhV3kwGks5siJdXgaJpZM4PW8Ba>
> .
>

@joelarmstrong revert which PR?
#1731, but actually I think I can get a proper fix in during the upcoming meeting. It's probably better to fix it properly than revert.
Actually, just noticed this has a slightly different stacktrace than #1858, though it has exactly the same cause, so reverting that PR wouldn't fix it anyway.

Resolving patch:
diff --git a/src/toil/batchSystems/options.py b/src/toil/batchSystems/options.py
--- a/src/toil/batchSystems/options.py
+++ b/src/toil/batchSystems/options.py
@@ -16,11 +16,27 @@
 from .registry import batchSystemFactoryFor, defaultBatchSystem, uniqueNames
 
 import socket
-
-
-def getLocalIP():
-    # may return localhost on some systems (not osx and coreos) https://stackoverflow.com/a/166520
-    return socket.gethostbyname(socket.gethostname())
+from contextlib import closing
+
+def getPublicIP():
+    """Get the IP that this machine uses to contact the internet.
+
+    If behind a NAT, this will still be this computer's IP, and not the router's."""
+    try:
+        # Try to get the internet-facing IP by attempting a connection
+        # to a non-existent server and reading what IP was used.
+        with closing(socket.socket(socket.AF_INET, socket.SOCK_DGRAM)) as sock:
+            # 203.0.113.0/24 is reserved as TEST-NET-3 by RFC 5737, so
+            # there is guaranteed to be no one listening on the other
+            # end (and we won't accidentally DOS anyone).
+            sock.connect(('203.0.113.1', 1))
+            ip = sock.getsockname()[0]
+        return ip
+    except:
+        # Something went terribly wrong. Just give loopback rather
+        # than killing everything, because this is often called just
+        # to provide a default argument
+        return '127.0.0.1'
 
 def _parasolOptions(addOptionFn):
     addOptionFn("--parasolCommand", dest="parasolCommand", default=None,
@@ -43,8 +59,8 @@ def _singleMachineOptions(addOptionFn):
 
 
 def _mesosOptions(addOptionFn):
-    addOptionFn("--mesosMaster", dest="mesosMasterAddress", default=None,
-                help=("The host and port of the Mesos master separated by colon. default=%s" % 'localhost:5050'))
+    addOptionFn("--mesosMaster", dest="mesosMasterAddress", default=getPublicIP() + ':5050',
+                help=("The host and port of the Mesos master separated by colon. (default: %(default)s)"))
 
 # Built in batch systems that have options
 _OPTIONS = [
@@ -96,7 +112,7 @@ def setDefaultOptions(config):
     config.linkImports = False
 
     # mesos
-    config.mesosMasterAddress = '%s:5050' % getLocalIP()
+    config.mesosMasterAddress = '%s:5050' % getPublicIP()
 
     # parasol
     config.parasolCommand = 'parasol'
diff --git a/src/toil/realtimeLogger.py b/src/toil/realtimeLogger.py
--- a/src/toil/realtimeLogger.py
+++ b/src/toil/realtimeLogger.py
@@ -25,13 +25,14 @@
 import json
 import logging
 import logging.handlers
-import socket
 import threading
 
 # Python 3 compatibility imports
 from six.moves import socketserver as SocketServer
 
 import toil.lib.bioio
+from toil.batchSystems.options import getPublicIP
+
 from future.utils import with_metaclass
 
 log = logging.getLogger(__name__)
@@ -145,16 +146,7 @@ def _startLeader(cls, batchSystem, level=defaultLevel):
                     cls.serverThread.start()
 
                     # Set options for logging in the environment so they get sent out to jobs
-                    fqdn = socket.getfqdn()
-                    try:
-                        ip = socket.gethostbyname(fqdn)
-                    except socket.gaierror:
-                        # FIXME: Does this only happen for me? Should we librarize the work-around?
-                        import platform
-                        if platform.system() == 'Darwin' and '.' not in fqdn:
-                            ip = socket.gethostbyname(fqdn + '.local')
-                        else:
-                            raise
+                    ip = getPublicIP()
                     port = cls.loggingServer.server_address[1]
 
                     def _setEnv(name, value):


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -960,7 +960,7 @@ def testSDBDomainsDeletedOnFailedJobstoreBucketCreation(self):
             testJobStoreUUID = str(uuid.uuid4())
             # Create the nucket at the external region
             s3 = S3Connection()
-            for attempt in retry_s3():
+            for attempt in retry_s3(delays=(2,5,10,30,60), timeout=600):
                 with attempt:
                     bucket = s3.create_bucket('domain-test-' + testJobStoreUUID + '--files',
                                               location=externalAWSLocation)