Repository: pantsbuild/pants

Issue:
Fixed overhead for python3 vs python2 in pantsbuild/pants
When comparing `./pants help` with/without `pants2`, there appears to be a fixed ~2 second overhead on my machine for python3.

Profiling using https://www.pantsbuild.org/invoking.html#profiling-pants doesn't show any significant difference between `pants2` and `pants`, so it's possible that this overhead exists below/before pants' own entrypoint...?
V2 rule resolution causing substantial startup performance hit at import time
### FYI - how to evaluate import time performance
Python 3.7 introduced a new performance profile for imports. Throughout this entire PR, to evaluate the performance, apply this diff:

```diff
diff --git a/pants b/pants
index 416d6e191..a2ed8d646 100755
--- a/pants
+++ b/pants
@@ -92,7 +92,7 @@ source ${HERE}/build-support/pants_venv
 source ${HERE}/build-support/bin/native/bootstrap_code.sh

 # Default to using Python 3 if not otherwise specified.
-export PY="${PY:-python3}"
+export PY="${PY:-python3.7}"

 # Set interpreter constraints to exactly match the interpreter used for the venv, if not already set.
 # Note that $PY only impacts which virtualenv we use for the parent Pants process; we must also set
@@ -133,7 +133,7 @@ function exec_pants_bare() {
   activate_pants_venv 1>&2
   bootstrap_native_code 1>&2
   PYTHONPATH="${PANTS_SRCPATH}:${PYTHONPATH}" \
-    exec python ${PANTS_EXE} "$@"
+    exec python -X importtime ${PANTS_EXE} "$@"
 }

 if [[ ! -z "${WRAPPER_REQUIREMENTS}" ]]; then
```

Then, when running Pants, run `./pants 2> my_log_name.log`

Finally, we can use the library `tuna` to analyze the results: 
1. `pip3 install tuna`
2. `tuna my_log_name.log`

We cannot use this profile with Python 2 or Python 3.6, but regardless we need to fix this slowdown.

### Original issue - Python 3 performance regression
We identified Python 3 has a fixed slowdown of about 2 seconds when running any command, e.g. `./pants` vs. `./pants2`. This overhead appears because of the line `module = importlib.import_module(module_path)`, which when running locally is approximately the same as `import pants.bin.pants_exe as module`. See https://github.com/pantsbuild/pants/issues/7373.

This import results in a whole 2.8 seconds.

![performance_pants-loader](https://user-images.githubusercontent.com/14852634/55089999-98652900-506b-11e9-8539-c641bf695093.png)

Notably, the import for `pants.backend.native.subsystems.native_toolchain` takes a whole 1.27 seconds!

### Issue appears to be `@rule` resolution
`pants.backend.native.subsystems.native_toolchain` seems to take so long because it uses the V2 engine, and at import time all of its function signatures, **including its `@rule` decorators**, must be resolved. Decorators are not lazily evaluated, and are evaluated at import time.

As a simple test for this, removing the rules-related imports for does result in a substantial speedup.

Apply this diff:
```diff
diff --git a/src/python/pants/backend/native/subsystems/xcode_cli_tools.py b/src/python/pants/backend/native/subsystems/xcode_cli_tools.py
index 3c2e47278..d8eb191ea 100644
--- a/src/python/pants/backend/native/subsystems/xcode_cli_tools.py
+++ b/src/python/pants/backend/native/subsystems/xcode_cli_tools.py
@@ -6,7 +6,6 @@ from __future__ import absolute_import, division, print_function, unicode_litera
 
 import os
 
-from pants.backend.native.config.environment import Assembler, CCompiler, CppCompiler, Linker
 from pants.engine.rules import rule
 from pants.engine.selectors import Select
 from pants.subsystem.subsystem import Subsystem
@@ -131,6 +130,7 @@ class XCodeCLITools(Subsystem):
 
   @memoized_method
   def assembler(self):
+    from pants.backend.native.config.environment import Assembler, CCompiler, CppCompiler, Linker
     return Assembler(
       path_entries=self.path_entries(),
       exe_filename='as',
@@ -139,6 +139,7 @@ class XCodeCLITools(Subsystem):
 
   @memoized_method
   def linker(self):
+    from pants.backend.native.config.environment import Assembler, CCompiler, CppCompiler, Linker
     return Linker(
       path_entries=self.path_entries(),
       exe_filename='ld',
@@ -150,6 +151,7 @@ class XCodeCLITools(Subsystem):
 
   @memoized_method
   def c_compiler(self):
+    from pants.backend.native.config.environment import Assembler, CCompiler, CppCompiler, Linker
     return CCompiler(
       path_entries=self.path_entries(),
       exe_filename='clang',
@@ -159,6 +161,7 @@ class XCodeCLITools(Subsystem):
 
   @memoized_method
   def cpp_compiler(self):
+    from pants.backend.native.config.environment import Assembler, CCompiler, CppCompiler, Linker
     return CppCompiler(
       path_entries=self.path_entries(),
       exe_filename='clang++',
@@ -167,30 +170,6 @@ class XCodeCLITools(Subsystem):
       extra_args=[MIN_OSX_VERSION_ARG])
 
 
-@rule(Assembler, [Select(XCodeCLITools)])
-def get_assembler(xcode_cli_tools):
-  return xcode_cli_tools.assembler()
-
-
-@rule(Linker, [Select(XCodeCLITools)])
-def get_ld(xcode_cli_tools):
-  return xcode_cli_tools.linker()
-
-
-@rule(CCompiler, [Select(XCodeCLITools)])
-def get_clang(xcode_cli_tools):
-  return xcode_cli_tools.c_compiler()
-
-
-@rule(CppCompiler, [Select(XCodeCLITools)])
-def get_clang_plusplus(xcode_cli_tools):
-  return xcode_cli_tools.cpp_compiler()
-
-
 def create_xcode_cli_tools_rules():
   return [
-    get_assembler,
-    get_ld,
-    get_clang,
-    get_clang_plusplus,
   ]
```

<img width="1662" alt="Xcode import without rules resolution" src="https://user-images.githubusercontent.com/14852634/55091365-ef6bfd80-506d-11e9-9d37-af56c4f2ff4c.png">


### Foil in this hypothesis: importing directly is fine
The above runs assume we are trying to run `bin/pants_loader.py`, which is what we normally do when running `./pants`. The `backend/native` code is only being imported as a dependency.

We can apply this diff to directly import `backend/native`:

```diff
diff --git a/pants b/pants
index a2ed8d646..ef074a23f 100755
--- a/pants
+++ b/pants
@@ -108,7 +108,7 @@ if [[ "${ONLY_USING_SINGLE_PYTHON_VERSION:-false}" != 'true' ]]; then
   clean_if_interpreter_constraints_changed "${PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS}"
 fi
 
-PANTS_EXE="${HERE}/src/python/pants/bin/pants_loader.py"
+PANTS_EXE="${HERE}/src/python/pants/backend/native/subsystems/native_toolchain.py"
 
 if [[ ! -z "${WRAPPER_REQUIREMENTS}" ]]; then
   REQUIREMENTS=(
```

When running `./pants`, we then get this perfectly reasonable import time. Why is this import 4 times faster than when `pants_loader.py` indirectly imports the same code?

<img width="1667" alt="Importing backend/native directly" src="https://user-images.githubusercontent.com/14852634/55090713-cf880a00-506c-11e9-8170-19240335425b.png">

### Other open questions
1. Why is this so much slower in Python 3 than Python 2. We know the difference is entirely from the speed of import time. Presumably, Python 2 is also evaluating the rules at import time - is there an even deeper issue that rules resolution is slower in Python 3?
1. How do we actually fix this? I don't know of any pattern to lazily evaluate decorators. Perhaps we can lazily import the relevant modules? Is there some way to cache the import?
 


Discussion:
Looking at https://speed.python.org/comparison/ and filtering down to the startup benchmarks. 
<img width="1372" alt="Screen Shot 2019-03-21 at 6 55 36 PM" src="https://user-images.githubusercontent.com/385657/54790094-54a59600-4c0b-11e9-92be-ca0303597be8.png">

It would seem that startup time for python has gone up in python 3. Not sure that explains 100% of the regression. 

It looks like the scale on that graph is seconds, so I don't think that python 3's inherent startup time could be causing this.

Given that https://www.pantsbuild.org/invoking.html#profiling-pants isn't showing anything useful, another possibility is that the overhead is outside of pants' own entrypoint somewhere... but it's likely that just adding some debug `print('in $x: {}'.format(time.time()))` lines in a bunch of spots would be sufficient to locate this (and then figure out why our profiling harness isn't catching it).
I've also updated the description: this repros for:
```
$ time ./pants2 help > /dev/null

real	0m1.952s
user	0m1.207s
sys	0m0.669s
```
vs
```
$ time ./pants help > /dev/null

real	0m4.221s
user	0m3.594s
sys	0m0.550s
```
Found one whole second of additional overhead due to [this line](https://github.com/pantsbuild/pants/blob/master/src/python/pants/bin/pants_loader.py#L104): `module = importlib.import_module(module_path)`.

Apply this diff:

```diff
diff --git a/src/python/pants/bin/pants_loader.py b/src/python/pants/bin/pants_loader.py
index 91a4ca161..cd39da454 100644
--- a/src/python/pants/bin/pants_loader.py
+++ b/src/python/pants/bin/pants_loader.py
@@ -7,6 +7,7 @@ from __future__ import absolute_import, division, print_function, unicode_litera
 import importlib
 import locale
 import os
+import time
 import sys
 import warnings
 from builtins import object
@@ -99,11 +100,23 @@ class PantsLoader(object):
 
   @staticmethod
   def load_and_execute(entrypoint):
+    time_before_load = time.time()
+    print("time before load: {}".format(time_before_load))
+
     assert ':' in entrypoint, 'ERROR: entrypoint must be of the form `module.path:callable`'
     module_path, func_name = entrypoint.split(':', 1)
+    time_after_parsing_entrypoint = time.time()
+    print("time after parsing entrypoint: {} (elapsed since last {})".format(time_after_parsing_entrypoint, time_after_parsing_entrypoint - time_before_load))
+
     module = importlib.import_module(module_path)
+    time_after_importing_module = time.time()
+    print("time after importing module: {} (elapsed since last {})".format(time_after_importing_module, time_after_importing_module - time_after_parsing_entrypoint))
+
     entrypoint_main = getattr(module, func_name)
     assert callable(entrypoint_main), 'ERROR: entrypoint `{}` is not callable'.format(entrypoint)
+    time_after_getting_entrypoint = time.time()
+    print("time after getting entrypoint: {} (elapsed since last {})".format(time_after_getting_entrypoint, time_after_getting_entrypoint - time_after_importing_module))
+
     entrypoint_main()
 
   @classmethod
```

Running `./pants2 help`:
```
time before load: 1553648263.19
time after parsing entrypoint: 1553648263.19 (elapsed since last 4.19616699219e-05)
time after importing module: 1553648263.79 (elapsed since last 0.609071016312)
time after getting entrypoint: 1553648263.79 (elapsed since last 2.00271606445e-05)
```

Running `./pants help`:
```
time before load: 1553648215.361344
time after parsing entrypoint: 1553648215.361392 (elapsed since last 4.792213439941406e-05)
time after importing module: 1553648217.008014 (elapsed since last 1.6466219425201416)
time after getting entrypoint: 1553648217.008049 (elapsed since last 3.504753112792969e-05)
```

--

Initial idea why this could be happening: as of Py3.3, `importlib.import_module` imports parent packages too: https://docs.python.org/3/library/importlib.html#importlib.import_module.

Also, this only explains half of the slow down.

> Initial idea why this could be happening: as of Py3.3, importlib.import_module imports parent packages too: https://docs.python.org/3/library/importlib.html#importlib.import_module.

I don't think this is the issue. Even with implementing our own import loader by following the tutorial from https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly and https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module, and intentionally leaving out the code to resolve parents, we still get an execution of 1.5 - 1.8 seconds due to the line `spec.loader.exec_module(module)`.

```diff
diff --git a/src/python/pants/bin/pants_loader.py b/src/python/pants/bin/pants_loader.py
index 91a4ca161..048c9e5fa 100644
--- a/src/python/pants/bin/pants_loader.py
+++ b/src/python/pants/bin/pants_loader.py
@@ -7,6 +7,7 @@ from __future__ import absolute_import, division, print_function, unicode_litera
 import importlib
 import locale
 import os
+import time
 import sys
 import warnings
 from builtins import object
@@ -97,13 +98,35 @@ class PantsLoader(object):
   def determine_entrypoint(env_var, default):
     return os.environ.pop(env_var, default)
 
+  @staticmethod
+  def load_module(name):
+    """An approximate implementation of import."""
+    spec = importlib.util.find_spec(name)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    sys.modules[name] = module
+    return module
+
   @staticmethod
   def load_and_execute(entrypoint):
+    time_before_load = time.time()
+    print("time before load: {}".format(time_before_load))
+
     assert ':' in entrypoint, 'ERROR: entrypoint must be of the form `module.path:callable`'
     module_path, func_name = entrypoint.split(':', 1)
-    module = importlib.import_module(module_path)
+    time_after_parsing_entrypoint = time.time()
+    print("time after parsing entrypoint: {} (elapsed since last {})".format(time_after_parsing_entrypoint, time_after_parsing_entrypoint - time_before_load))
+
+    # module = importlib.import_module(module_path)
+    module = PantsLoader.load_module(module_path)
+    time_after_importing_module = time.time()
+    print("time after importing module: {} (elapsed since last {})".format(time_after_importing_module, time_after_importing_module - time_after_parsing_entrypoint))
+
     entrypoint_main = getattr(module, func_name)
     assert callable(entrypoint_main), 'ERROR: entrypoint `{}` is not callable'.format(entrypoint)
+    time_after_getting_entrypoint = time.time()
+    print("time after getting entrypoint: {} (elapsed since last {})".format(time_after_getting_entrypoint, time_after_getting_entrypoint - time_after_importing_module))
+
     entrypoint_main()
 
   @classmethod
```

So, the issue is `exec_module()`. I don't know why this would be slower in Py3 and am not sure how we could fix it.

Going to take a break from the investigation.
Also note this issue reproduces with simply `./pants` vs `./pants2`.

```
$ time ./pants

user 3.91s
sys 0.46s
```
vs
```
$ time ./pants2

user 1.11s
sys 0.55s
```

@stuhood you may want to update the issue description.
Distilling down observations a bit:

Re: import / exec_module - these are the same thing for our purposes - a detail of importing is the fact that it means to execute a module in a new namespace. This is true in python2 and python3 and we really have no control over that - the language requires imports work this way.

Narrowing the testing above and further eliminating the resolution of parents red-herring by switching to `__import__`:
```diff
$ git diff
diff --git a/src/python/pants/bin/pants_loader.py b/src/python/pants/bin/pants_loader.py
index 91a4ca161..c987228ec 100644
--- a/src/python/pants/bin/pants_loader.py
+++ b/src/python/pants/bin/pants_loader.py
@@ -4,7 +4,6 @@
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-import importlib
 import locale
 import os
 import sys
@@ -101,7 +100,13 @@ class PantsLoader(object):
   def load_and_execute(entrypoint):
     assert ':' in entrypoint, 'ERROR: entrypoint must be of the form `module.path:callable`'
     module_path, func_name = entrypoint.split(':', 1)
-    module = importlib.import_module(module_path)
+
+    import time
+    start = time.time()
+    __import__(module_path)
+    module = sys.modules[module_path]
+    print('>>> import took {}'.format(time.time() - start))
+
     entrypoint_main = getattr(module, func_name)
     assert callable(entrypoint_main), 'ERROR: entrypoint `{}` is not callable'.format(entrypoint)
     entrypoint_main()

```

py3:
```
$ time ./pants -V
>>> import took 1.0852248668670654
1.15.0rc0

real	0m3.074s
user	0m2.675s
sys	0m0.283s
```

py2
```
$ time ./pants2 -V
>>> import took 0.541138887405
1.15.0rc0

real	0m1.512s
user	0m1.181s
sys	0m0.227s
```

-V executes as little code as possible so the overall times here represent almost pure importing. We measure one bit of that - the time it takes to import the pants entrypoint - and this takes 2x as long under python3. We already know from historical work that a large portion of pants start up time is consumed by gathering options from Tasks and Subsystems. This involves importing more code outside of what we measured here. I think it's safe to say then that import being 2x slower explains everything here.

Figuring out why import is 2x slower remains. Is it something we have control over or not?


Eeinteresting - helpful new python3.7 feature:
```
$ PYTHONPROFILEIMPORTTIME=1 ./pants -V &> /tmp/import-times.txt
```
Yields: [import-times.txt](https://github.com/pantsbuild/pants/files/3014461/import-times.txt)


Via `grep "import time:" /tmp/import-times.txt | cut -d: -f2 | sort -n | sed -re "s/\|\s+/| /g" | tail -10`
```
self [us] | cumulative | imported package
...
     42707 | 42707 | pants.engine.isolated_process
     47359 | 415113 | pants.backend.python.rules.python_test_runner
    101299 | 101461 | pants.backend.native.subsystems.binaries.gcc
    107934 | 108030 | pants.backend.native.subsystems.binaries.binutils
    108567 | 182715 | pants.engine.legacy.options_parsing
    151087 | 151087 | pants.backend.native.subsystems.binaries.llvm
    182072 | 182072 | pants.engine.build_files
    208188 | 208188 | pants.backend.native.subsystems.xcode_cli_tools
    362700 | 366863 | pants.engine.legacy.graph
    550027 | 1119320 | pants.backend.native.subsystems.native_toolchain
```

The interesting times here are in the left-hand column - they give the time to import the module itself. The heaviest hitter takes 500ms and their are several 100ms+. 

Also interesting is comparing `@rule` counts:
```
$ git grep @rule src/**/*.py | cut -d: -f1 | sort | uniq -c | sort -n | tail -10
      2 src/python/pants/backend/native/subsystems/binaries/binutils.py
      2 src/python/pants/backend/native/subsystems/binaries/gcc.py
      3 src/python/pants/backend/native/subsystems/binaries/llvm.py
      3 src/python/pants/engine/legacy/options_parsing.py
      3 src/python/pants/init/engine_initializer.py
      4 src/python/pants/backend/native/subsystems/xcode_cli_tools.py
      4 src/python/pants/engine/build_files.py
      7 src/python/pants/engine/legacy/graph.py
     12 src/python/pants/engine/rules.py
     14 src/python/pants/backend/native/subsystems/native_toolchain.py
```

The ratios seem to track the import self time ratios well.

Perhaps putting an even finer point on things is looking at the tree output of the profile combined with the diff above for emphasis:
```
$ PYTHONPROFILEIMPORTTIME=1 ./pants -V
...
import time:       636 |    1019874 |       pants.pantsd.pants_daemon
import time:       300 |    1030937 |     pants.bin.remote_pants_runner
import time:       216 |    1031152 |   pants.bin.pants_runner
import time:       165 |    1114329 | pants.bin.pants_exe
>>> import took 1.1143436431884766
import time:       182 |        182 | pants.build_graph.aliased_target
import time:       177 |        177 | pants.build_graph.intransitive_dependency
import time:       206 |        206 | pants.build_graph.prep_command
import time:       216 |        216 |   netrc
...
import time:       133 |        133 | pants.backend.python.targets.python_requirement_library
import time:       224 |        224 |     pants.build_graph.import_remote_sources_mixin
import time:       212 |        436 |   pants.backend.python.targets.import_wheels_mixin
import time:       225 |        660 | pants.backend.python.targets.unpacked_whls
import time:       122 |        122 |   pants.backend.python.tasks
import time:        98 |         98 |       pants.backend.native
import time:       159 |        256 |     pants.backend.native.targets
import time:        94 |         94 |       pants.backend.native.subsystems
import time:        93 |         93 |         pants.backend.native.subsystems.utils
import time:       215 |        308 |       pants.backend.native.subsystems.utils.mirrored_target_option_mixin
import time:       143 |        143 |       pants.option.compiler_option_sets_mixin
import time:       687 |       1231 |     pants.backend.native.subsystems.native_build_step
import time:       287 |        287 |     pants.backend.native.targets.native_artifact
import time:       316 |       2088 |   pants.backend.native.targets.native_library
import time:       135 |        135 |     pants.backend.native.tasks
import time:        97 |         97 |       pants.backend.native.config
import time:      2152 |       2249 |     pants.backend.native.config.environment
import time:       141 |        141 |         pants.backend.native.subsystems.native_build_settings
import time:        98 |         98 |             pants.backend.native.subsystems.binaries
import time:    109589 |     109687 |           pants.backend.native.subsystems.binaries.binutils
import time:       175 |        175 |             pants.backend.native.subsystems.utils.archive_file_mapper
import time:    102889 |     103064 |           pants.backend.native.subsystems.binaries.gcc
import time:    153892 |     153892 |           pants.backend.native.subsystems.binaries.llvm
import time:       257 |        257 |             pants.backend.native.subsystems.utils.parse_search_dirs
import time:       415 |        671 |           pants.backend.native.subsystems.libc_dev
import time:    205049 |     205049 |           pants.backend.native.subsystems.xcode_cli_tools
import time:    525163 |    1097523 |         pants.backend.native.subsystems.native_toolchain
import time:       242 |        242 |         pants.backend.native.targets.packaged_native_library
import time:       156 |        156 |         pants.build_graph.dependency_context
import time:       367 |    1098427 |       pants.backend.native.tasks.native_task
import time:       951 |    1099377 |     pants.backend.native.tasks.native_compile
import time:       687 |    1102447 |   pants.backend.native.tasks.link_shared_libraries
import time:       156 |        156 |       pex.compiler
import time:       365 |        521 |     pex.pex_builder
import time:       272 |        272 |     pants.backend.python.subsystems.python_setup
import time:       452 |       1244 |   pants.backend.python.subsystems.pex_build_util
import time:       287 |        287 |         pex.environment
import time:       374 |        661 |       pex.pex
import time:       176 |        836 |     pants.binaries.executable_pex_tool
import time:       342 |       1178 |   pants.backend.python.subsystems.python_native_code
import time:       465 |    1107543 | pants.backend.python.tasks.build_local_python_distributions
import time:       227 |        227 | pants.backend.python.tasks.gather_sources
import time:       124 |        124 |     pants.backend.python.subsystems.python_tool_base
import time:       151 |        274 |   pants.backend.python.subsystems.isort
import time:      1069 |       1069 |     pants.backend.python.interpreter_cache
import time:       287 |       1356 |   pants.backend.python.tasks.python_tool_prep_base
import time:       215 |       1844 | pants.backend.python.tasks.isort_prep
import time:       241 |        241 | pants.backend.python.tasks.isort_run
import time:       135 |        135 |   pants.util.fileutil
...
import time:       448 |      18731 | pants.contrib.python.checks.tasks.checkstyle.checkstyle
import time:       314 |        314 | pants.contrib.python.checks.tasks.python_eval
import time:        87 |         87 |   pants.contrib.scrooge.tasks
import time:       144 |        144 |   pants.contrib.scrooge.tasks.java_thrift_library_fingerprint_strategy
import time:       300 |        300 |   pants.contrib.scrooge.tasks.thrift_util
import time:       702 |       1232 | pants.contrib.scrooge.tasks.scrooge_gen
import time:       273 |        273 | pants.contrib.scrooge.tasks.thrift_linter
import time:       149 |        149 |   pants.contrib.thrifty.java_thrifty_library
import time:       205 |        354 | pants.contrib.thrifty.java_thrifty_gen
import time:       215 |        215 |   multiprocessing.dummy.connection
import time:       313 |        527 | multiprocessing.dummy
```

It really does look like something about the `@rule` decorator code is very slow to execute on import in general which is now doubly slow under py3.



I don't believe "rule resolution" is likely to be involved in the slowdown here -- `@rule` is just doing `ast` parsing, see: https://github.com/pantsbuild/pants/blob/37654aa9b0e123f36ac9efdba75e05d5f26b8042/src/python/pants/engine/rules.py#L232-L233 Actually compiling the rules and checking correctness happens after import time, here: https://github.com/pantsbuild/pants/blob/37654aa9b0e123f36ac9efdba75e05d5f26b8042/src/python/pants/engine/scheduler.py#L85 and a bit in rust when creating the scheduler below that as well.

The slowness of import when doing anything nontrivial at the top level e.g. parsing `@rule`s is part of why the daemon is such a necessary work stream, but the difference compared to py2 isn't. If the py3 import profiler also lets us drill down into individual methods, that would allow us to tell whether the `ast.parse()` call or the ast traversal is causing the majority of the import time issues (which would help us determine what changed from py2 to py3 and how to fix it). We may be able to do this with the normal pex profiler or something if we e.g. import `pants.util.contextutil.maybe_profiled` first, then wrap an import that pulls in all our rules with that.
@cosmicexplorer see: https://github.com/pantsbuild/pants/issues/7373#issuecomment-477258388 which seems to definitely point to @rule or the typical code using @rule today.

Resolving patch:
diff --git a/src/python/pants/engine/rules.py b/src/python/pants/engine/rules.py
--- a/src/python/pants/engine/rules.py
+++ b/src/python/pants/engine/rules.py
@@ -9,6 +9,7 @@
 import inspect
 import itertools
 import logging
+import sys
 from abc import abstractproperty
 from builtins import bytes, str
 from types import GeneratorType
@@ -34,14 +35,13 @@
 class _RuleVisitor(ast.NodeVisitor):
   """Pull `Get` calls out of an @rule body and validate `yield` statements."""
 
-  def __init__(self, func, func_node, func_source, orig_indent, frame, parents_table):
+  def __init__(self, func, func_node, func_source, orig_indent, parents_table):
     super(_RuleVisitor, self).__init__()
     self._gets = []
     self._func = func
     self._func_node = func_node
     self._func_source = func_source
     self._orig_indent = orig_indent
-    self._frame = frame
     self._parents_table = parents_table
     self._yields_in_assignments = set()
 
@@ -51,7 +51,8 @@ def gets(self):
 
   def _generate_ast_error_message(self, node, msg):
     # This is the location info of the start of the decorated @rule.
-    filename, line_number, _, context_lines, _ = inspect.getframeinfo(self._frame, context=4)
+    filename = inspect.getsourcefile(self._func)
+    source_lines, line_number = inspect.getsourcelines(self._func)
 
     # The asttokens library is able to keep track of line numbers and column offsets for us -- the
     # stdlib ast library only provides these relative to each parent node.
@@ -82,14 +83,14 @@ def _generate_ast_error_message(self, node, msg):
 
 The rule defined by function `{func_name}` begins at:
 {filename}:{line_number}:{orig_indent}
-{context_lines}
+{source_lines}
 """.format(func_name=self._func.__name__, msg=msg,
            filename=filename, line_number=line_number, orig_indent=self._orig_indent,
            node_line_number=node_file_line,
            node_col=fully_indented_node_col,
            node_text=indented_node_text,
            # Strip any leading or trailing newlines from the start of the rule body.
-           context_lines=''.join(context_lines).strip('\n')))
+           source_lines=''.join(source_lines).strip('\n')))
 
   class YieldVisitError(Exception): pass
 
@@ -242,7 +243,7 @@ def wrapper(func):
     if not inspect.isfunction(func):
       raise ValueError('The @rule decorator must be applied innermost of all decorators.')
 
-    caller_frame = inspect.stack()[1][0]
+    owning_module = sys.modules[func.__module__]
     source = inspect.getsource(func)
     beginning_indent = _get_starting_indent(source)
     if beginning_indent:
@@ -250,9 +251,13 @@ def wrapper(func):
     module_ast = ast.parse(source)
 
     def resolve_type(name):
-      resolved = caller_frame.f_globals.get(name) or caller_frame.f_builtins.get(name)
-      if not isinstance(resolved, type):
-        raise ValueError('Expected a `type` constructor, but got: {}'.format(name))
+      resolved = getattr(owning_module, name, None) or owning_module.__builtins__.get(name, None)
+      if resolved is None:
+        raise ValueError('Could not resolve type `{}` in top level of module {}'
+                         .format(name, owning_module.__name__))
+      elif not isinstance(resolved, type):
+        raise ValueError('Expected a `type` constructor for `{}`, but got: {} (type `{}`)'
+                         .format(name, resolved, type(resolved).__name__))
       return resolved
 
     gets = OrderedSet()
@@ -270,7 +275,6 @@ def resolve_type(name):
       func_node=rule_func_node,
       func_source=source,
       orig_indent=beginning_indent,
-      frame=caller_frame,
       parents_table=parents_table,
     )
     rule_visitor.visit(rule_func_node)


Test patch:
diff --git a/tests/python/pants_test/engine/test_rules.py b/tests/python/pants_test/engine/test_rules.py
--- a/tests/python/pants_test/engine/test_rules.py
+++ b/tests/python/pants_test/engine/test_rules.py
@@ -60,6 +60,9 @@ def __repr__(self):
 _suba_root_rules = [RootRule(SubA)]
 
 
+_this_is_not_a_type = 3
+
+
 @console_rule('example', [Select(Console)])
 def a_console_rule_generator(console):
   a = yield Get(A, str('a str!'))
@@ -81,7 +84,7 @@ def test_creation_fails_with_bad_declaration_type(self):
       RuleIndex.create([A()])
 
 
-class RuleGraphTest(unittest.TestCase):
+class RuleGraphTest(TestBase):
   def test_ruleset_with_missing_product_type(self):
     @rule(A, [Select(B)])
     def a_from_b_noop(b):
@@ -694,7 +697,24 @@ def b_from_d(d):
                      }""").strip(),
                                     subgraph)
 
-  def test_validate_yield_statements(self):
+  def test_invalid_get_arguments(self):
+    with self.assertRaisesWithMessage(ValueError, """\
+Could not resolve type `XXX` in top level of module pants_test.engine.test_rules"""):
+      class XXX(object): pass
+      @rule(A, [])
+      def f():
+        a = yield Get(A, XXX, 3)
+        yield a
+
+    # This fails because the argument is defined in this file's module, but it is not a type.
+    with self.assertRaisesWithMessage(ValueError, """\
+Expected a `type` constructor for `_this_is_not_a_type`, but got: 3 (type `int`)"""):
+      @rule(A, [])
+      def g():
+        a = yield Get(A, _this_is_not_a_type, 3)
+        yield a
+
+  def test_validate_yield_statements_in_rule_body(self):
     with self.assertRaisesRegexp(_RuleVisitor.YieldVisitError, re.escape('yield A()')):
       @rule(A, [])
       def f():
@@ -737,12 +757,14 @@ def g():
 
 The rule defined by function `g` begins at:
 test_rules.py:{rule_lineno}:{rule_col}
-    with self.assertRaises(_RuleVisitor.YieldVisitError) as cm:
       @rule(A, [])
       def g():
-""".format(lineno=(sys._getframe().f_lineno - 20),
+        # This is a yield statement without an assignment, and not at the end.
+        yield Get(B, D, D())
+        yield A()
+""".format(lineno=(sys._getframe().f_lineno - 22),
            col=8,
-           rule_lineno=(sys._getframe().f_lineno - 25),
+           rule_lineno=(sys._getframe().f_lineno - 27),
            rule_col=6))
 
   def create_full_graph(self, rules, validate=True):