Repository: pantsbuild/pants

Issue:
When pantsd is enabled, the jvm executor env scrubbing seems to not work
To reproduce, change the pants.ini compile.zinc jvm_options to include `'-Xms4g',` then `export _JAVA_OPTIONS=-Xmx4g` and try to compile anything. You should see 

```
compile(...) failed: Failed to read nailgun output after 10.0 seconds!
```

If you run `clean-all` and then re-run with `--no-enable-pantsd`, the compile will succeed.


Discussion:
I wasn't able to repro this... In the pants repo:

```
$ git diff
diff --git a/pants.ini b/pants.ini
index 5e390db..e83fd7d 100644
--- a/pants.ini
+++ b/pants.ini
@@ -186,7 +186,7 @@ exclude_patterns: [
 
 [compile.zinc]
 jvm_options: [
-    '-Xmx4g', '-XX:+UseConcMarkSweepGC', '-XX:ParallelGCThreads=4',
+    '-Xms4g', '-XX:+UseConcMarkSweepGC', '-XX:ParallelGCThreads=4',
   ]
 
 args: [

$ env | egrep -i 'pantsd|_java'
PANTS_ENABLE_PANTSD=True
_JAVA_OPTIONS=-Xmx4g
$ ./pants run examples/src/java/org/pantsbuild/example/hello/simple

18:06:59 00:00 [main]
               (To run a reporting server: ./pants server)
18:07:00 00:01   [setup]
18:07:00 00:01     [parse]
               Executing tasks in goals: jvm-platform-validate -> bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> resolve -> resources -> compile -> pyprep -> binary -> run
18:07:01 00:02   [jvm-platform-validate]
18:07:01 00:02     [jvm-platform-validate]
18:07:01 00:02   [bootstrap]
18:07:01 00:02     [substitute-aliased-targets]
18:07:01 00:02     [jar-dependency-management]
18:07:01 00:02     [bootstrap-jvm-tools]
18:07:01 00:02     [provide-tools-jar]
18:07:01 00:02   [imports]
18:07:01 00:02     [ivy-imports]
18:07:01 00:02   [unpack-jars]
18:07:01 00:02     [unpack-jars]
18:07:01 00:02   [deferred-sources]
18:07:01 00:02     [deferred-sources]
18:07:01 00:02   [gen]
18:07:01 00:02     [antlr-java]
18:07:01 00:02     [antlr-py]
18:07:01 00:02     [jaxb]
18:07:01 00:02     [protoc]
18:07:01 00:02     [ragel]
18:07:01 00:02     [thrift-java]
18:07:01 00:02     [thrift-py]
18:07:01 00:02     [wire]
18:07:01 00:02     [avro-java]
18:07:01 00:02     [go-thrift]
18:07:01 00:02     [go-protobuf]
18:07:01 00:02     [jax-ws]
18:07:01 00:02     [scrooge]
18:07:01 00:02     [thrifty]
18:07:01 00:02   [resolve]
18:07:01 00:02     [ivy]
18:07:01 00:02     [coursier]
18:07:01 00:02     [go]
18:07:01 00:02     [scala-js-compile]
18:07:01 00:02     [scala-js-link]
18:07:01 00:02     [node]
18:07:01 00:02   [resources]
18:07:01 00:02     [prepare]
18:07:01 00:02     [services]
18:07:01 00:02   [compile]
18:07:01 00:02     [node]
18:07:01 00:02     [compile-jvm-prep-command]
18:07:01 00:02       [jvm_prep_command]
18:07:01 00:02     [compile-prep-command]
18:07:01 00:02     [compile]
18:07:01 00:02     [zinc]
18:07:01 00:02     [javac]
18:07:01 00:02     [cpp]
18:07:01 00:02     [errorprone]
18:07:01 00:02     [findbugs]
18:07:01 00:02     [go]
18:07:02 00:03   [pyprep]
18:07:02 00:03     [interpreter]
18:07:02 00:03     [build-local-dists]
18:07:02 00:03     [requirements]
18:07:02 00:03     [sources]
18:07:02 00:03   [binary]
18:07:02 00:03     [binary-jvm-prep-command]
18:07:02 00:03       [jvm_prep_command]
18:07:02 00:03     [binary-prep-command]
18:07:02 00:03     [py]
18:07:02 00:03     [py-wheels]
18:07:02 00:03     [jvm]
                   creating dist/simple.jar
18:07:02 00:03       [create-monolithic-jar]
18:07:02 00:03         [add-internal-classes]
18:07:02 00:03         [add-dependency-jars]
18:07:02 00:03     [dup]
18:07:02 00:03     [cpplib]
18:07:02 00:03       [cpp-library]
18:07:02 00:03     [cpp]
18:07:02 00:03       [cpp-binary]
18:07:02 00:03     [go]
18:07:02 00:03   [run]
18:07:02 00:03     [py]
18:07:02 00:03     [jvm]
18:07:02 00:03       [run]
WARN] Scrubbing _JAVA_OPTIONS=-Xmx4g
Picked up _JAVA_OPTIONS: -Xmx4g
Hello World!

18:07:02 00:03     [cpp]
18:07:02 00:03     [go]
18:07:02 00:03     [node]
               Waiting for background workers to finish.
18:07:02 00:03   [complete]
               SUCCESS
```

Anything obviously wrong with my repro attempt?
It looks like you are reproing it--see `Picked up _JAVA_OPTIONS: -Xmx4g`, but it doesn't cause a crash for some reason. I may have the wrong set of flags to trigger the JVM crash.
I think the JVM crash happens when `-Xmx` is < `-Xms`, because, as it explains, you can't have a starting heap that's larger than the max heap.

```
$ java -Xmx4g -Xms5g
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size
```

Yep, so `mx` == `ms` works.
I still don't reproduce a crash, even setting Xmx < Xms...

```
$ export _JAVA_OPTIONS=-Xmx3g
$ ps aux | grep nailgun | awk '{print $2}' | xargs kill
kill: 77970: No such process
$ ./pants kill-pantsd

18:42:14 00:00 [main]
               (To run a reporting server: ./pants server)
18:42:14 00:00   [setup]
18:42:14 00:00     [parse]
               Executing tasks in goals: kill-pantsd
18:42:14 00:00   [kill-pantsd]
18:42:14 00:00     [kill-pantsd]
18:42:15 00:01   [complete]
               SUCCESS
$ ./pants run examples/src/java/org/pantsbuild/example/hello/simple

18:42:18 00:00 [main]
               (To run a reporting server: ./pants server)
18:42:22 00:04   [setup]
18:42:22 00:04     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> jvm-platform-validate -> pyprep -> resolve -> resources -> compile -> binary -> run
18:42:22 00:04   [bootstrap]
18:42:22 00:04     [substitute-aliased-targets]
18:42:22 00:04     [jar-dependency-management]
18:42:22 00:04     [bootstrap-jvm-tools]
18:42:22 00:04     [provide-tools-jar]
18:42:22 00:04   [imports]
18:42:22 00:04     [ivy-imports]
18:42:22 00:04   [unpack-jars]
18:42:22 00:04     [unpack-jars]
18:42:23 00:05   [deferred-sources]
18:42:23 00:05     [deferred-sources]
18:42:23 00:05   [gen]
18:42:23 00:05     [antlr-java]
18:42:23 00:05     [antlr-py]
18:42:23 00:05     [jaxb]
18:42:23 00:05     [protoc]
18:42:23 00:05     [ragel]
18:42:23 00:05     [thrift-java]
18:42:23 00:05     [thrift-py]
18:42:23 00:05     [wire]
18:42:23 00:05     [avro-java]
18:42:23 00:05     [go-thrift]
18:42:23 00:05     [jax-ws]
18:42:23 00:05     [scrooge]
18:42:23 00:05     [thrifty]
18:42:23 00:05   [jvm-platform-validate]
18:42:23 00:05     [jvm-platform-validate]
18:42:23 00:05   [pyprep]
18:42:23 00:05     [interpreter]
18:42:23 00:05     [build-local-dists]
18:42:23 00:05     [requirements]
18:42:23 00:05     [sources]
18:42:23 00:05   [resolve]
18:42:23 00:05     [ivy]
18:42:23 00:05     [coursier]
18:42:23 00:05     [go]
18:42:23 00:05     [scala-js-compile]
18:42:23 00:05     [scala-js-link]
18:42:23 00:05     [node]
18:42:23 00:05   [resources]
18:42:23 00:05     [prepare]
18:42:23 00:05     [services]
18:42:23 00:05   [compile]
18:42:23 00:05     [node]
18:42:23 00:05     [compile-jvm-prep-command]
18:42:23 00:05       [jvm_prep_command]
18:42:23 00:05     [compile-prep-command]
18:42:23 00:05     [compile]
18:42:23 00:05     [zinc]
18:42:23 00:05     [javac]
18:42:23 00:05     [jvm-dep-check]
18:42:23 00:05     [cpp]
18:42:23 00:05     [errorprone]
18:42:23 00:05     [findbugs]
18:42:23 00:05     [go]
18:42:23 00:05   [binary]
18:42:23 00:05     [binary-jvm-prep-command]
18:42:23 00:05       [jvm_prep_command]
18:42:23 00:05     [binary-prep-command]
18:42:23 00:05     [py]
18:42:23 00:05     [py-wheels]
18:42:23 00:05     [jvm]
                   creating dist/simple.jar
18:42:23 00:05       [create-monolithic-jar]
18:42:23 00:05         [add-internal-classes]
18:42:23 00:05         [add-dependency-jars]WARN] Scrubbing _JAVA_OPTIONS=-Xmx3g

18:42:24 00:06     [dup]
18:42:24 00:06     [cpplib]
18:42:24 00:06       [cpp-library]
18:42:24 00:06     [cpp]
18:42:24 00:06       [cpp-binary]
18:42:24 00:06     [go]
18:42:24 00:06   [run]
18:42:24 00:06     [py]
18:42:24 00:06     [jvm]
18:42:24 00:06       [run]
WARN] Scrubbing _JAVA_OPTIONS=-Xmx3g
Picked up _JAVA_OPTIONS: -Xmx3g
Hello World!

18:42:24 00:06     [cpp]
18:42:24 00:06     [go]
18:42:24 00:06     [node]
               Waiting for background workers to finish.
18:42:24 00:06   [complete]
               SUCCESS
```

Resolving patch:
diff --git a/src/python/pants/util/collections.py b/src/python/pants/util/collections.py
--- a/src/python/pants/util/collections.py
+++ b/src/python/pants/util/collections.py
@@ -9,3 +9,13 @@
 def combined_dict(*dicts):
   """Combine one or more dicts into a new, unified dict (dicts to the right take precedence)."""
   return {k: v for d in dicts for k, v in d.items()}
+
+
+def recursively_update(d, d2):
+  """dict.update but which merges child dicts (dict2 takes precedence where there's conflict)."""
+  for k, v in d2.items():
+    if k in d:
+      if isinstance(v, dict):
+        recursively_update(d[k], v)
+        continue
+    d[k] = v


Test patch:
diff --git a/tests/python/pants_test/pantsd/test_pantsd_integration.py b/tests/python/pants_test/pantsd/test_pantsd_integration.py
--- a/tests/python/pants_test/pantsd/test_pantsd_integration.py
+++ b/tests/python/pants_test/pantsd/test_pantsd_integration.py
@@ -17,7 +17,7 @@
 from concurrent.futures import ThreadPoolExecutor
 
 from pants.pantsd.process_manager import ProcessManager
-from pants.util.collections import combined_dict
+from pants.util.collections import recursively_update
 from pants.util.contextutil import environment_as, temporary_dir
 from pants.util.dirutil import rm_rf, safe_file_dump, safe_mkdir, touch
 from pants_test.pants_run_integration_test import PantsResult, PantsRunIntegrationTest
@@ -96,15 +96,17 @@ def pantsd_test_context(self, log_level='info', extra_config=None, expected_runs
         workdir = os.path.join(workdir_base, '.workdir.pants.d')
         print('\npantsd log is {}/pantsd/pantsd.log'.format(workdir))
         pantsd_config = {
-          'GLOBAL': combined_dict({
+          'GLOBAL': {
             'enable_pantsd': True,
             # The absolute paths in CI can exceed the UNIX socket path limitation
             # (>104-108 characters), so we override that here with a shorter path.
             'watchman_socket_path': '/tmp/watchman.{}.sock'.format(os.getpid()),
             'level': log_level,
             'pants_subprocessdir': pid_dir,
-          }, extra_config or {})
+          }
         }
+        if extra_config:
+          recursively_update(pantsd_config, extra_config)
         print('>>> config: \n{}\n'.format(pantsd_config))
         checker = PantsDaemonMonitor(pid_dir)
         self.assert_success_runner(workdir, pantsd_config, ['kill-pantsd'])
@@ -124,13 +126,14 @@ def pantsd_test_context(self, log_level='info', extra_config=None, expected_runs
           checker.assert_stopped()
 
   @contextmanager
-  def pantsd_successful_run_context(self, log_level='info', extra_config=None):
+  def pantsd_successful_run_context(self, log_level='info', extra_config=None, extra_env=None):
     with self.pantsd_test_context(log_level, extra_config) as (workdir, pantsd_config, checker):
       yield (
         functools.partial(
           self.assert_success_runner,
           workdir,
-          pantsd_config
+          pantsd_config,
+          extra_env=extra_env,
         ),
         checker,
         workdir,
@@ -144,16 +147,18 @@ def _run_count(self, workdir):
     else:
       return 0
 
-  def assert_success_runner(self, workdir, config, cmd, extra_config={}, expected_runs=1):
-    combined_config = combined_dict(config, extra_config)
-    print(bold(cyan('\nrunning: ./pants {} (config={})'
-                    .format(' '.join(cmd), combined_config))))
+  def assert_success_runner(self, workdir, config, cmd, extra_config={}, extra_env={}, expected_runs=1):
+    combined_config = config.copy()
+    recursively_update(combined_config, extra_config)
+    print(bold(cyan('\nrunning: ./pants {} (config={}) (extra_env={})'
+                    .format(' '.join(cmd), combined_config, extra_env))))
     run_count = self._run_count(workdir)
     start_time = time.time()
     run = self.run_pants_with_workdir(
       cmd,
       workdir,
       combined_config,
+      extra_env=extra_env,
       # TODO: With this uncommented, `test_pantsd_run` fails.
       # tee_output=True
     )
@@ -180,10 +185,12 @@ def test_pantsd_compile(self):
 
   def test_pantsd_run(self):
     extra_config = {
+      'GLOBAL': {
         # Muddies the logs with warnings: once all of the warnings in the repository
         # are fixed, this can be removed.
         'glob_expansion_failure': 'ignore',
       }
+    }
     with self.pantsd_successful_run_context(
           'debug',
           extra_config=extra_config
@@ -408,7 +415,7 @@ def test_pantsd_launch_env_var_is_not_inherited_by_pantsd_runner_children(self):
 
   def test_pantsd_invalidation_file_tracking(self):
     test_file = 'testprojects/src/python/print_env/main.py'
-    config = {'pantsd_invalidation_globs': '["testprojects/src/python/print_env/*"]'}
+    config = {'GLOBAL': {'pantsd_invalidation_globs': '["testprojects/src/python/print_env/*"]'}}
     with self.pantsd_successful_run_context(extra_config=config) as (
       pantsd_run, checker, workdir, _
     ):
@@ -528,3 +535,16 @@ def test_pantsd_multiple_parallel_runs(self):
       waiter_result = PantsResult(waiter_pants_command, waiter_pants_process.returncode, waiter_stdout_data.decode("utf-8"),
         waiter_stderr_data.decode("utf-8"), workdir)
       self.assert_success(waiter_result)
+
+  def test_pantsd_environment_scrubbing(self):
+    # This pair of JVM options causes the JVM to always crash, so the command will fail if the env
+    # isn't stripped.
+    with self.pantsd_successful_run_context(
+      extra_config={'compile.zinc': {'jvm_options': ['-Xmx1g']}},
+      extra_env={'_JAVA_OPTIONS': '-Xms2g'},
+    ) as (pantsd_run, checker, workdir, _):
+      pantsd_run(['help'])
+      checker.assert_started()
+
+      result = pantsd_run(['compile', 'examples/src/java/org/pantsbuild/example/hello/simple'])
+      self.assert_success(result)
diff --git a/tests/python/pants_test/util/test_collections.py b/tests/python/pants_test/util/test_collections.py
--- a/tests/python/pants_test/util/test_collections.py
+++ b/tests/python/pants_test/util/test_collections.py
@@ -7,7 +7,7 @@
 
 import unittest
 
-from pants.util.collections import combined_dict
+from pants.util.collections import combined_dict, recursively_update
 
 
 class TestCollections(unittest.TestCase):
@@ -20,3 +20,10 @@ def test_combined_dict(self):
       ),
       {'a': 1, 'b': 2, 'c': 3}
     )
+
+  def test_recursively_update(self):
+    d = {'a': 1, 'b': {'c': 2, 'o': 'z'}, 'z': {'y': 0}}
+    recursively_update(d, {'e': 3, 'b': {'f': 4, 'o': 9}, 'g': {'h': 5}, 'z': 7})
+    self.assertEqual(
+      d, {'a': 1, 'b': {'c': 2, 'f': 4, 'o': 9}, 'e': 3, 'g': {'h': 5}, 'z': 7}
+    )