Repository: pantsbuild/pants

Issue:
initial pantsd startup fails with watchman timeout in larger repos due to watchman startup lag
in our monorepo, starting pantsd typically fails on the first attempt with a timeout:

```
I0131 10:29:15.079124 39004 pants_daemon.py:164] pantsd starting, log level is DEBUG
I0131 10:29:15.081336 39004 pants_daemon.py:125] setting up service <pants.pantsd.service.fs_event_service.FSEventService object at 0x107ba7c90>
I0131 10:29:15.081749 39004 pants_daemon.py:125] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x107ccd910>
I0131 10:29:15.082030 39004 pants_daemon.py:125] setting up service <pants.pantsd.service.pailgun_service.PailgunService object at 0x107ccd350>
I0131 10:29:15.082367 39004 pants_daemon.py:138] starting service <pants.pantsd.service.fs_event_service.FSEventService object at 0x107ba7c90>
I0131 10:29:15.082813 39004 pants_daemon.py:138] starting service <pants.pantsd.service.pailgun_service.PailgunService object at 0x107ccd350>
I0131 10:29:15.083915 39004 pailgun_service.py:96] starting pailgun server on port 49648
I0131 10:29:15.084602 39004 pants_daemon.py:138] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x107ccd910>
W0131 10:29:20.093113 39004 pants_daemon.py:34] Exception in thread Thread-4:
W0131 10:29:20.094105 39004 pants_daemon.py:34] Traceback (most recent call last):
W0131 10:29:20.094321 39004 pants_daemon.py:34]   File "/opt/twitter_mde/package/python2.7/current/lib/python2.7/threading.py", line 810, in __bootstrap_inner
W0131 10:29:20.094489 39004 pants_daemon.py:34]     self.run()
W0131 10:29:20.094674 39004 pants_daemon.py:34]   File "/opt/twitter_mde/package/python2.7/current/lib/python2.7/threading.py", line 763, in run
W0131 10:29:20.095588 39004 pants_daemon.py:34]     self.__target(*self.__args, **self.__kwargs)
W0131 10:29:20.095909 39004 pants_daemon.py:34]   File "/Users/kwilson/dev/source/.pex/install/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl.48dc014d73cb0c2ada6c0a2b629cc338a0d5f764/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl/pants/pantsd/service/fs_event_service.py", line 108, in run
W0131 10:29:20.096282 39004 pants_daemon.py:34]     self._watchman.watch_project(self._build_root)
W0131 10:29:20.097439 39004 pants_daemon.py:34]   File "/Users/kwilson/dev/source/.pex/install/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl.48dc014d73cb0c2ada6c0a2b629cc338a0d5f764/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl/pants/pantsd/watchman.py", line 140, in watch_project
W0131 10:29:20.097753 39004 pants_daemon.py:34]     return self.client.query('watch-project', os.path.realpath(path))
W0131 10:29:20.098042 39004 pants_daemon.py:34]   File "/Users/kwilson/dev/source/.pex/install/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl.33236a849596954b95b805adeef6af661ee59480/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl/pywatchman/__init__.py", line 492, in query
W0131 10:29:20.099023 39004 pants_daemon.py:34]     res = self.receive()
W0131 10:29:20.099283 39004 pants_daemon.py:34]   File "/Users/kwilson/dev/source/.pex/install/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl.33236a849596954b95b805adeef6af661ee59480/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl/pywatchman/__init__.py", line 430, in receive
W0131 10:29:20.099884 39004 pants_daemon.py:34]     result = self.recvConn.receive()
W0131 10:29:20.100061 39004 pants_daemon.py:34]   File "/Users/kwilson/dev/source/.pex/install/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl.33236a849596954b95b805adeef6af661ee59480/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl/pywatchman/__init__.py", line 272, in receive
W0131 10:29:20.100225 39004 pants_daemon.py:34]     buf = [self.transport.readBytes(sniff_len)]
W0131 10:29:20.100390 39004 pants_daemon.py:34]   File "/Users/kwilson/dev/source/.pex/install/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl.33236a849596954b95b805adeef6af661ee59480/pywatchman-1.3.0-cp27-cp27m-macosx_10_11_intel.whl/pywatchman/__init__.py", line 173, in readBytes
W0131 10:29:20.100619 39004 pants_daemon.py:34]     raise SocketTimeout('timed out waiting for response')
W0131 10:29:20.101134 39004 pants_daemon.py:34] SocketTimeout: timed out waiting for response
I0131 10:29:21.087065 39004 pants_daemon.py:82] terminating pantsd service: <pants.pantsd.service.fs_event_service.FSEventService object at 0x107ba7c90>
I0131 10:29:21.087440 39004 fs_event_service.py:49] shutting down threadpool
I0131 10:29:21.088057 39004 pants_daemon.py:82] terminating pantsd service: <pants.pantsd.service.pailgun_service.PailgunService object at 0x107ccd350>
W0131 10:29:21.088500 39004 pailgun_service.py:104] pailgun service shutting down
I0131 10:29:21.089570 39004 pants_daemon.py:82] terminating pantsd service: <pants.pantsd.service.scheduler_service.SchedulerService object at 0x107ccd910>
I0131 10:29:21.099920 39004 pants_daemon.py:85] terminating pantsd
F0131 10:29:21.101201 39004 process_manager.py:435] Traceback (most recent call last):
  File "/Users/kwilson/dev/source/.pex/install/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl.48dc014d73cb0c2ada6c0a2b629cc338a0d5f764/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl/pants/pantsd/process_manager.py", line 433, in daemonize
    self.post_fork_child(**post_fork_child_opts or {})
  File "/Users/kwilson/dev/source/.pex/install/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl.48dc014d73cb0c2ada6c0a2b629cc338a0d5f764/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl/pants/pantsd/pants_daemon.py", line 188, in post_fork_child
    self._run()
  File "/Users/kwilson/dev/source/.pex/install/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl.48dc014d73cb0c2ada6c0a2b629cc338a0d5f764/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl/pants/pantsd/pants_daemon.py", line 177, in _run
    self._run_services(self._services)
  File "/Users/kwilson/dev/source/.pex/install/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl.48dc014d73cb0c2ada6c0a2b629cc338a0d5f764/pantsbuild.pants-1.3.0.dev7+2468485184-py2-none-any.whl/pants/pantsd/pants_daemon.py", line 150, in _run_services
    raise self.RuntimeFailure('service failure for {}, shutting down!'.format(service))
RuntimeFailure: service failure for <pants.pantsd.service.fs_event_service.FSEventService object at 0x107ba7c90>, shutting down!
```

this seems to be largely due to blocking activity in the watchman daemon, which fails to respond within the client timeout period. subsequent pantsd startup runs are successful as the watchman daemon (which is still running in the face of a pantsd failure) has had time to process all of the filesystem information before the subsequent startup attempt.


Discussion:

Resolving patch:
diff --git a/src/python/pants/pantsd/subsystem/watchman_launcher.py b/src/python/pants/pantsd/subsystem/watchman_launcher.py
--- a/src/python/pants/pantsd/subsystem/watchman_launcher.py
+++ b/src/python/pants/pantsd/subsystem/watchman_launcher.py
@@ -31,6 +31,9 @@ def register_options(cls, register):
       register('--supportdir', advanced=True, default='bin/watchman',
                help='Find watchman binaries under this dir. Used as part of the path to lookup '
                     'the binary with --binary-util-baseurls and --pants-bootstrapdir.')
+      register('--startup-timeout', type=float, advanced=True, default=Watchman.STARTUP_TIMEOUT_SECONDS,
+               help='The watchman socket timeout (in seconds) for the initial `watch-project` command. '
+                    'This may need to be set higher for larger repos due to watchman startup cost.')
       register('--socket-timeout', type=float, advanced=True, default=Watchman.SOCKET_TIMEOUT_SECONDS,
                help='The watchman client socket timeout (in seconds).')
       register('--socket-path', type=str, advanced=True, default=None,
@@ -45,11 +48,12 @@ def create(self):
                               options.level,
                               options.version,
                               options.supportdir,
+                              options.startup_timeout,
                               options.socket_timeout,
                               options.socket_path)
 
   def __init__(self, binary_util, workdir, log_level, watchman_version, watchman_supportdir,
-               socket_timeout, socket_path_override=None):
+               startup_timeout, socket_timeout, socket_path_override=None):
     """
     :param binary_util: The BinaryUtil subsystem instance for binary retrieval.
     :param workdir: The current pants workdir.
@@ -63,6 +67,7 @@ def __init__(self, binary_util, workdir, log_level, watchman_version, watchman_s
     self._workdir = workdir
     self._watchman_version = watchman_version
     self._watchman_supportdir = watchman_supportdir
+    self._startup_timeout = startup_timeout
     self._socket_timeout = socket_timeout
     self._socket_path_override = socket_path_override
     self._log_level = log_level
@@ -86,6 +91,7 @@ def watchman(self):
     return Watchman(watchman_binary,
                     self._workdir,
                     self._convert_log_level(self._log_level),
+                    self._startup_timeout,
                     self._socket_timeout,
                     self._socket_path_override)
 
diff --git a/src/python/pants/pantsd/watchman.py b/src/python/pants/pantsd/watchman.py
--- a/src/python/pants/pantsd/watchman.py
+++ b/src/python/pants/pantsd/watchman.py
@@ -20,16 +20,18 @@
 class Watchman(ProcessManager):
   """Watchman process manager and helper class."""
 
+  STARTUP_TIMEOUT_SECONDS = 30.0
   SOCKET_TIMEOUT_SECONDS = 5.0
 
   EventHandler = namedtuple('EventHandler', ['name', 'metadata', 'callback'])
 
-  def __init__(self, watchman_path, work_dir, log_level='1', timeout=SOCKET_TIMEOUT_SECONDS,
-               socket_path_override=None, metadata_base_dir=None):
+  def __init__(self, watchman_path, work_dir, log_level='1', startup_timeout=STARTUP_TIMEOUT_SECONDS,
+               timeout=SOCKET_TIMEOUT_SECONDS, socket_path_override=None, metadata_base_dir=None):
     """
     :param str watchman_path: The path to the watchman binary.
     :param str work_dir: The path to the pants work dir.
-    :param float timeout: The watchman socket timeout (in seconds).
+    :param float startup_timeout: The timeout for the initial `watch-project` query (in seconds).
+    :param float timeout: The watchman socket timeout for all subsequent queries (in seconds).
     :param str log_level: The watchman log level. Watchman has 3 log levels: '0' for no logging,
                           '1' for standard logging and '2' for verbose logging.
     :param str socket_path_override: The overridden target path of the watchman socket, if any.
@@ -42,6 +44,7 @@ def __init__(self, watchman_path, work_dir, log_level='1', timeout=SOCKET_TIMEOU
     self._watchman_path = self._normalize_watchman_path(watchman_path)
     self._watchman_work_dir = os.path.join(work_dir, self.name)
     self._log_level = log_level
+    self._startup_timeout = startup_timeout
     self._timeout = timeout
 
     self._state_file = os.path.join(self._watchman_work_dir, '{}.state'.format(self.name))
@@ -60,7 +63,10 @@ def client(self):
 
   def _make_client(self):
     """Create a new watchman client using the BSER protocol over a UNIX socket."""
-    return StreamableWatchmanClient(sockpath=self.socket, transport='local', timeout=self._timeout)
+    self._logger.debug('setting initial watchman timeout to %s', self._startup_timeout)
+    return StreamableWatchmanClient(sockpath=self.socket,
+                                    transport='local',
+                                    timeout=self._startup_timeout)
 
   def _is_valid_executable(self, binary_path):
     return os.path.isfile(binary_path) and os.access(binary_path, os.X_OK)
@@ -137,7 +143,12 @@ def watch_project(self, path):
 
     :param string path: the path to the watchman project root/pants build root.
     """
-    return self.client.query('watch-project', os.path.realpath(path))
+    # TODO(kwlzn): Add a client.query(timeout=X) param to the upstream pywatchman project.
+    try:
+      return self.client.query('watch-project', os.path.realpath(path))
+    finally:
+      self.client.setTimeout(self._timeout)
+      self._logger.debug('setting post-startup watchman timeout to %s', self._timeout)
 
   def subscribed(self, build_root, handlers):
     """Bulk subscribe generator for StreamableWatchmanClient.


Test patch: