puppetlabs/puppet — pull request #753 by nicklewis (2012-05-10 18:50:11+00:00)
Title: Significantly improve compilation performance when using modules

When autoloading classes/defines, the typeloader constructs a set of
possible locations for the class, based on its name. Effectively, it
will look in the canonical locations corresponding to each namespace in
the fully-qualified name. So for each namespace, it will ask the
environment for a Puppet::Module instance for that module, to ask it
which of the module's potentially manifests match the class it's looking
for. To answer that request, the environment instantiates a
Puppet::Module.

This amounts to potentially thousands of Puppet::Module instances being
created, because it does this many times (based on nesting of the class
name) per include/autoload/import. When Puppet::Module instances are
created, they parse and load their metadata.json file, in part to
validate their use. This implies that each compilation results in
metadata.json being parsed thousands of times, which is extremely slow
(and obviously provides no actual benefit).

Fortunately, the environment object already keeps a list of
Puppet::Module instances for every module in its modulepath. The fix
applied here is simply to change the environment such that it provides
modules by looking them up in its cached list, resulting in up to an
order of magnitude improvement in compilation time.