Repository: pantsbuild/pants

Issue:
Using pantsd in multiple workspaces results in watchman crashes
If you use `pantsd` in multiple workspaces, the startup of the daemon in one workspace will cause watchman in the other workspace to crash with:
```
W1218 11:12:33.162800 41529 pants_daemon.py:50] Traceback (most recent call last):
W1218 11:12:33.163083 41529 pants_daemon.py:50]   File "/opt/ee/python/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
W1218 11:12:33.163443 41529 pants_daemon.py:50]     self.run()
W1218 11:12:33.163623 41529 pants_daemon.py:50]   File "/opt/ee/python/2.7/lib/python2.7/threading.py", line 763, in run
W1218 11:12:33.163882 41529 pants_daemon.py:50]     self.__target(*self.__args, **self.__kwargs)
W1218 11:12:33.164051 41529 pants_daemon.py:50]   File "/Users/stuhood/.pex/install/pantsbuild.pants-1.4.0.dev16+811811700-cp27-none-macosx_10_8_x86_64.whl.f06e58a5054ed2f4674a13da4dee0c9606177989/pantsbuild.pants-1.4.0.dev16+811811700-cp27-none-macosx_10_8_x86_64.whl/pants/pantsd/service/fs_event_service.py", line 116, in run
W1218 11:12:33.164232 41529 pants_daemon.py:50]     for handler_name, event_data in self._watchman.subscribed(self._build_root, subscriptions):
W1218 11:12:33.164388 41529 pants_daemon.py:50]   File "/Users/stuhood/.pex/install/pantsbuild.pants-1.4.0.dev16+811811700-cp27-none-macosx_10_8_x86_64.whl.f06e58a5054ed2f4674a13da4dee0c9606177989/pantsbuild.pants-1.4.0.dev16+811811700-cp27-none-macosx_10_8_x86_64.whl/pants/pantsd/watchman.py", line 188, in subscribed
W1218 11:12:33.164566 41529 pants_daemon.py:50]     raise self.WatchmanCrash(e)
W1218 11:12:33.164716 41529 pants_daemon.py:50] WatchmanCrash: empty watchman response
```

Probably some watchman path is global, but shouldn't be.


Discussion:
almost certainly a problem of the watchman build itself, as manually starting watchman will trigger a cross-workspace crash:

```
[omerta pants (master)]$ ps -ef |grep 67473
  501 67473     1   0  5:27PM ??         0:11.58 /Users/kwilson/.cache/pants/bin/watchman/mac/10.12/4.5.0/watchman --foreground --logfile=/Users/kwilson/dev/source/.pids/watchman/watchman.log --log-level=1 --sockname=/Users/kwilson/dev/source/.pids/watchman/watchman.sock --statefile=/Users/kwilson/dev/source/.pids/watchman/watchman.state
  501 68106 11529   0  5:27PM ttys003    0:00.00 grep 67473
[omerta pants (master)]$ /Users/kwilson/.cache/pants/bin/watchman/mac/10.12/4.5.0/watchman get-pid --no-save-state --statefile=/Users/kwilson/dev/pants/.pids/watchman/watchman.state --sockname=/Users/kwilson/dev/pants/.pids/watchman/watchman.sock --logfile=/Users/kwilson/dev/pants/.pids/watchman/watchman.log --log-level 1
{
    "version": "4.5.0",
    "pid": 68406
}
[omerta pants (master)]$ ps -ef |grep 67473
  501 68459 11529   0  5:27PM ttys003    0:00.00 grep 67473
[omerta pants (master)]$
```

watchman logs indicate a signal 15, but from the `init` process?

```
Terminating due to signal 15 Terminated generated by pid=1 uid=0.  (0x0)
0   watchman                            0x00000001030f8c2d crash_handler + 365
1   libsystem_platform.dylib            0x00007fffe3106b3a _sigtramp + 26
2   ???                                 0x000000010eea7978 0x0 + 4545214840
3   watchman                            0x0000000103109558 run_service + 216
4   watchman                            0x00000001031093c7 main + 2679
5   libdyld.dylib                       0x00007fffe2ef7235 start + 1
6   ???                                 0x0000000000000006 0x0 + 6
```
and fwict, an OSX-specific problem. I'm able to launch watchman in two different workspaces on Linux just fine:

```
[root@2c781dbf6e16 pants2]# ps -ef |grep watchman
root      5980     1  0 01:52 ?        00:00:00 /root/.cache/pants/bin/watchman/linux/x86_64/4.5.0/watchman get-pid --no-save-state --statefile=/show/pants/.pids/watchman/watchman.state --sockname=/show/pants/.pids/watchman/watchman.sock --logfile=/show/pants/.pids/watchman/watchman.log --log-level 1
root      6163     1  1 01:57 ?        00:00:00 /root/.cache/pants/bin/watchman/linux/x86_64/4.5.0/watchman get-pid --no-save-state --statefile=/show/pants2/.pids/watchman/watchman.state --sockname=/show/pants2/.pids/watchman/watchman.sock --logfile=/show/pants2/.pids/watchman/watchman.log --log-level 1
```

(which is unfortunate, because `strace`)
and still happens after https://github.com/pantsbuild/pants/pull/5238
reproing by executing alternating watchman commands in disparate repos seems to be writing to `~/Library/LaunchAgents/com.github.facebook.watchman.plist` with different plist args each time.
filed an upstream issue here: https://github.com/facebook/watchman/issues/558
plan is to cut a new watchman binary if/when https://github.com/facebook/watchman/pull/559 lands upstream - but we should be able to cut a hotfixed bin whenever we like.

Resolving patch:
diff --git a/src/python/pants/option/global_options.py b/src/python/pants/option/global_options.py
--- a/src/python/pants/option/global_options.py
+++ b/src/python/pants/option/global_options.py
@@ -192,7 +192,7 @@ def register_bootstrap_options(cls, register):
              help='The number of workers to use for the filesystem event service executor pool.')
 
     # Watchman options.
-    register('--watchman-version', advanced=True, default='4.9.0', help='Watchman version.')
+    register('--watchman-version', advanced=True, default='4.9.0-pants1', help='Watchman version.')
     register('--watchman-supportdir', advanced=True, default='bin/watchman',
              help='Find watchman binaries under this dir. Used as part of the path to lookup '
                   'the binary with --binary-util-baseurls and --pants-bootstrapdir.')
diff --git a/src/python/pants/pantsd/watchman.py b/src/python/pants/pantsd/watchman.py
--- a/src/python/pants/pantsd/watchman.py
+++ b/src/python/pants/pantsd/watchman.py
@@ -51,6 +51,7 @@ def __init__(self, watchman_path, metadata_base_dir, log_level='1', startup_time
 
     self._state_file = os.path.join(self._watchman_work_dir, '{}.state'.format(self.name))
     self._log_file = os.path.join(self._watchman_work_dir, '{}.log'.format(self.name))
+    self._pid_file = os.path.join(self._watchman_work_dir, '{}.pid'.format(self.name))
     self._sock_file = socket_path_override or os.path.join(self._watchman_work_dir,
                                                            '{}.sock'.format(self.name))
 
@@ -83,10 +84,12 @@ def _maybe_init_metadata(self):
     # Initialize watchman with an empty, but valid statefile so it doesn't complain on startup.
     safe_file_dump(self._state_file, '{}')
 
-  def _construct_cmd(self, cmd_parts, state_file, sock_file, log_file, log_level):
+  def _construct_cmd(self, cmd_parts, state_file, sock_file, pid_file, log_file, log_level):
     return [part for part in cmd_parts] + ['--no-save-state',
+                                           '--no-site-spawner',
                                            '--statefile={}'.format(state_file),
                                            '--sockname={}'.format(sock_file),
+                                           '--pidfile={}'.format(pid_file),
                                            '--logfile={}'.format(log_file),
                                            '--log-level', log_level]
 
@@ -111,6 +114,7 @@ def launch(self):
     cmd = self._construct_cmd((self._watchman_path, 'get-pid'),
                               state_file=self._state_file,
                               sock_file=self._sock_file,
+                              pid_file=self._pid_file,
                               log_file=self._log_file,
                               log_level=str(self._log_level))
     self._logger.debug('watchman cmd is: {}'.format(' '.join(cmd)))


Test patch:
diff --git a/tests/python/pants_test/pantsd/test_watchman.py b/tests/python/pants_test/pantsd/test_watchman.py
--- a/tests/python/pants_test/pantsd/test_watchman.py
+++ b/tests/python/pants_test/pantsd/test_watchman.py
@@ -69,6 +69,7 @@ def test_construct_cmd(self):
     output = self.watchman._construct_cmd(['cmd', 'parts', 'etc'],
                                           'state_file',
                                           'sock_file',
+                                          'pid_file',
                                           'log_file',
                                           'log_level')
 
@@ -76,8 +77,10 @@ def test_construct_cmd(self):
                                'parts',
                                'etc',
                                '--no-save-state',
+                               '--no-site-spawner',
                                '--statefile=state_file',
                                '--sockname=sock_file',
+                               '--pidfile=pid_file',
                                '--logfile=log_file',
                                '--log-level',
                                'log_level'])