Implement a JSON file terminus to mirror YAML.

For a bunch of local caching and retrieval, Puppet uses a YAML backed local
file terminus to store, retrieve, and search content.  This is great, but the
YAML encoder is super-slow and painfully odd across Ruby versions.

Our PSON support is much more reliable, as well as substantially faster, and
is pretty much as capable.  There isn't much reason, other than migration
cost, to prefer YAML to PSON encoding of data transfer documents.

Having a JSON terminus allows the user to configure, manually, the use of JSON
everywhere that we previously used YAML.  While this doesn't bring the
benefits out of the box, it allows users to select this while we flush out any
lingering bugs left in the support code and system.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Add catalog JSON terminus to improve performance.

This adds a JSON backed catalog terminus, so that the catalog cache can
migrate away from YAML to a faster and more stable storage format.

This is not the default: that still falls to old, trustable YAML which we know
to be stable and effective.

To enable this, edit your routes.yaml file and add:

    agent:
      catalog:
        cache: json

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Implement Symbol#intern when it is missing.

In order to reduce use of `Puppet::Util#symbolize` in the code - which is
called some 90,000 times in a real-world run - being able to intern a symbol
is convenient.

This is a simple implementation: it just returns itself.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Use `intern` rather than `symbolize` it Puppet::Type

The cost of symbolize is ~ 50 percent higher than the cost of intern, when
applied to either a string or a symbol.  Given this is called some 90,000
times in my real world test run, that adds up to a non-zero amount of
wasted time.

Sadly, this only works out at 0.15 seconds real time reduction, but it is
better than nothing - and takes out 85,000 extra calls.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
zaml: statically determine the to_ascii8bit implementation.

Instead of asking every single string the same question, "do you respond to
`encoding`", we can just figure that out once and choose an implementation
based on the result - since it is a static test on the same class every time.

This saves ~ 0.1 second of runtime, which is small but cheap.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
zaml: unify on a single definition of `to_zaml` for Symbol

There were two competitive definitions of the `to_zaml` function over Symbol,
one in monkey_patches and one in the core ZAML library.

The monkey patch version won in practice, so now they are unified, and the
monkey patch version is adopted to replace the core ZAML version, in the zaml
Ruby file.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Merge branch 'feature/2.7.x/json-terminus-backport' of https://github.com/daniel-pittman/puppet into 2.7.x

* 'feature/2.7.x/json-terminus-backport' of https://github.com/daniel-pittman/puppet:
  Add catalog JSON terminus to improve performance.
  Implement a JSON file terminus to mirror YAML.
Merge branch 'perf/2.7.x/use-intern-not-symbolize' of https://github.com/daniel-pittman/puppet into 2.7.x

* 'perf/2.7.x/use-intern-not-symbolize' of https://github.com/daniel-pittman/puppet:
  Use `intern` rather than `symbolize` it Puppet::Type
  Implement Symbol#intern when it is missing.
Maint: Fix intermittently failing spec test

The spec test was only failing in the jenkins environment due to the
File.open stub interferring with the file cleanup routines. This commit
changes the test to raise on the readlock method instead, which
restricts its affect to just that example.
zaml: rework strings for correctness and speed

This changes the implementation of string output in the ZAML encoder.  On my
test that delivers around a fifty percent reduction in output time, from 4.11
to 1.89 seconds on my test of ~ 35,000 output strings.

It also adds a bunch of tests to validate the correctness of binary output
handling, and some test inputs derived from various security related UTF-8
encoding problems to validate that (sadly) we pass them through as
binary data.

Optimally we would treat binary and text input as distinct in the parser and
compiler of Puppet, then only allow legal UTF-8 in strings - but that is not
the world in which we live.

It finally fixes a bug in the binary output along the way - which, thankfully,
nobody actually seems to have run into since I introduced it in ea0dd1483fc2.

Because the Ruby BASE64 encoder will split lines at the 60 character mark, a
long binary input would end up with embedded newlines in places that were not
legal for YAML.

Instead of just emitting that, we run through the string encoder over the
transformed data, which leads to a solid output format that works entirely
as expected.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>

Conflicts:
	lib/puppet/util/zaml.rb
	spec/unit/util/zaml_spec.rb

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
zaml: add whitespace for readability.

The ZAML code has very little whitespace, leading to a difficult to read set
of code.  This just adds a little whitespace, with no behaviour change, to
improve that before I start hacking directly on that code.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
zaml: avoid calling emit when we don't have anything to output

The `nl` method allowed an additional string to be output after the newline;
this is seldom used, and the default implementation resulted in emitting an
empty string.

We can just skip that, for a minor win in function call count.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
zaml: don't use backreferences on strings

While the YAML spec requires that we emit a backreference if the same string
object occurs twice in our content, it also costs a lot.  About two seconds of
time encoding the output, in my testing, out of a total of three.

Everything we do internally with YAML doesn't actually *depend* on the ability
to serialize an object graph that treats String objects as duplicated.

This is reinforced by every object that can serialize to PSON - since
it *can't* have a back-reference, you automatically get string duplication
there.

That means that we can just cut those few seconds out of the code without any
particular cost or suffering, and shave *way* more time off a run than almost
anything else we can do to our YAML output code.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
zaml: handle multi-line map keys and array values correctly.

The ZAML encoder had some bugs when handling multi-line string values in map
keys and arrays. It also changed behaviour slightly when I modified it for
speed, causing a regression in another case that was correctly handled.

This fixes that, as well as making more robust the process of tagging
structural keys that can span multiple lines.

It also adds some tests to validate that we don't regress again.

This fixes a pre-existing bug in the encoder, as well as my regression.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Disable ActiveSupport::Dependencies hooks.

The ActiveSupport::Dependencies code is loaded along with Rails, and hooks
into a bunch of the system.  This consumes a surprisingly large amount of CPU
time watching for constants and generally acting to support unloading code.

This is related to the code that automatically requires libraries when the
constant is first accessed.

We don't - can't, since Rails is a soft dependencies - depend on that feature,
or anything related to it, we have no benefit from paying that horrible price.

Dropping this is actually a significant drop in CPU consumption - some 8
percent of the CPU cost in my testing is attributed directly to this feature.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Merge branch 'perf/2.7.x/backport-activesupport-hook-disabling' of https://github.com/daniel-pittman/puppet into 2.7.x

* 'perf/2.7.x/backport-activesupport-hook-disabling' of https://github.com/daniel-pittman/puppet:
  Disable ActiveSupport::Dependencies hooks.
Merge branch 'perf/2.7.x/backport-zaml-symbol-output-unification' of https://github.com/daniel-pittman/puppet into 2.7.x

* 'perf/2.7.x/backport-zaml-symbol-output-unification' of https://github.com/daniel-pittman/puppet:
  zaml: unify on a single definition of `to_zaml` for Symbol

Conflicts:
	lib/puppet/util/monkey_patches.rb
Merge branch 'perf/2.7.x/backport-zaml-performance-improvements' of https://github.com/daniel-pittman/puppet into 2.7.x

* 'perf/2.7.x/backport-zaml-performance-improvements' of https://github.com/daniel-pittman/puppet:
  zaml: handle multi-line map keys and array values correctly.
  zaml: don't use backreferences on strings
  zaml: avoid calling emit when we don't have anything to output
  zaml: add whitespace for readability.
  zaml: rework strings for correctness and speed
  zaml: statically determine the to_ascii8bit implementation.
Merge remote-tracking branch 'upstream/2.7.x' into 3.x

* upstream/2.7.x:
  Disable ActiveSupport::Dependencies hooks.
  zaml: handle multi-line map keys and array values correctly.
  zaml: don't use backreferences on strings
  zaml: avoid calling emit when we don't have anything to output
  zaml: add whitespace for readability.
  zaml: rework strings for correctness and speed
  Maint: Fix intermittently failing spec test
  zaml: unify on a single definition of `to_zaml` for Symbol
  zaml: statically determine the to_ascii8bit implementation.
  Use `intern` rather than `symbolize` it Puppet::Type
  Implement Symbol#intern when it is missing.
  Add catalog JSON terminus to improve performance.
  Implement a JSON file terminus to mirror YAML.

Conflicts:
	lib/puppet/feature/rails.rb
	lib/puppet/indirector/json.rb
	lib/puppet/type.rb
	lib/puppet/util/zaml.rb
	spec/unit/indirector/json_spec.rb
	spec/unit/util/zaml_spec.rb

The zaml code in 3.x did not handle multi-line values. This was fixed in
the backport to 2.7.x, and fixed in 2.7.x as part of this merge.

Also the rails feature assumed `require 'active_support'` would also
autoload the dependencies, but this does not occur with version 3.2.8.
This was fixed in 2.7.x (to explicitly require the dependencies) and
fixed in 3.x as part of this merge.
Merge remote-tracking branch 'upstream/3.x'

* upstream/3.x:
  Disable ActiveSupport::Dependencies hooks.
  zaml: handle multi-line map keys and array values correctly.
  zaml: don't use backreferences on strings
  zaml: avoid calling emit when we don't have anything to output
  zaml: add whitespace for readability.
  zaml: rework strings for correctness and speed
  Maint: Fix intermittently failing spec test
  zaml: unify on a single definition of `to_zaml` for Symbol
  zaml: statically determine the to_ascii8bit implementation.
  Use `intern` rather than `symbolize` it Puppet::Type
  Implement Symbol#intern when it is missing.
  Add catalog JSON terminus to improve performance.
  Implement a JSON file terminus to mirror YAML.