(PUP-1362) (PUP-1775) Use yum check-updates for latest yum packages

Early versions of the yum package provider used `yum list available` to
determine the latest version of packages that were `ensure => latest`.
However the versions of yum available around that time changed the
output formatting of yum, which meant that the parsing code would break
on a regular basis (#836).

Commit 8722e439 resolved this by using the yum python bindings to
interact with yum. This required shipping a python helper script to
provide the interface between Puppet and the python libraries. However
this opened up a host of other issues ranging from what version of
Python is symlinked to '/usr/bin/python', how the yum database is
locked, and so on (PUP-1362, #11380, #16623). In addition changes to
yumhelper.py were very hard to test or reason about since it doesn't
neatly plug into our existing tests.

This commit removes yumhelper.py and replaces the update checks by
calling `yum check-update`. The output format of this command has not
changed since CentOS 4 and works all the way up to RedHat 7 so we should
be able to rely on this for a while.
(PUP-3302) Fix resource when using directory environments

Before this fix when `puppet resource file` was being used set with
source defined puppet would error out because we need an environment
object in Puppet::Indirector::Request#environment= and instead of the
none environment being passed around it is passed as the string "none"
which is a directory environment that does not exist. This is what is
raising the EnvironmentNotFound error in puppet.

Instead of passing around a string we pass around the none environment
to fix the issue.
(PUP-3479) Update Windows ffi dependency to new ffi

Ruby 2.1.3 on Windows requires ffi 1.9.5. Update our Windows dependencies
to allow 1.9.5+ to be selected.
(PUP-3302) Fix Puppet Resource

@jpartlow and I paired on this issue.
Before this commit when using puppet resource with any resource that
went through the ral and needed a real environment would fail since the
default behaviour of Puppet::Resource::Catalog#initialize was to create
a new catalog used the None environment. In places where we were
performing `Puppet.lookup(:environments).get` on the None environment
puppet would fail, causing the none environment not found error to be
raised.

This commit changes the default behavior of Puppet::Resource::Ral#save
to create a catalog with the `current_environment` that is setup at
application initialization instead of the none environment. It also
makes `default_file_terminus` set to `file_server` in
`Puppet::Application::Resource` to make `puppet resource` act like
`puppet apply`.

We wanted to make sure that this change was not going to harm any other
parts of the puppet codebase and at least in puppet the ral is only used
when using `puppet resource`. We do not believe that it will have any
adverse effects anywhere else.
(PUP-914) Use resource ref when writing dot files

Use the `ref` attribute of a resource instead of `to_s`.

Internally, Puppet uses the `ref` attribute to identify and locate resources
within a catalog. This patch makes use of `ref` to identify nodes in graphviz
output as some Types, notably Whit, override `to_s` and either destroy or
obscure the identity of resource nodes.
(PUP-2706) Change adduser to useradd in postinst

There has been an intermittent error where the installation of
puppet-common is failing on Ubtuntu 14.04. This error is
```
chage: the shadow password file is not present
/usr/bin/chage failed with return code 15, shadow not enabled, password aging cannot be set. Continuing.
chfn: PAM: Authentication failure
adduser: `/usr/bin/chfn -f Puppet configuration management daemon puppet' returned error code 1. Exiting.
```

Though `chage` is complaining about the fact that shadow is not enabled,
it looks like the installation is failing sure to the `adduser` call.
The thought is that `adduser` does some things under the hood that
require PAM to be configured in the build environment, which it isn't.
If we change the call from `adduser` to `useradd`, hopefully we will not
run into this issue. Note: it is difficult to test this due to the
intermittent nature of the issue.

Also, since we're removing the use of `adduser`, we no longer need to
ensure that package is available on the system.
(PUP-3505) Fix problem with passing explicit undef res parameter

With the future parser in effect and a resource attribute is undef the
logic that checks for missing arguments did not report the parameter.
This in turn led to that the logic that either fails the operation, or
assigns the default value to completely skip that parameter/argument.
Subsequently a reference of that parameter ended up picking the same
variable from an outer context.

The fix is to equate a nil value (4x) with an :undef value (for 3x) when
computing the missing arguments.

A test is added that tests the functionality.
(PUP-3500) Avoid cacheing environment values when handling setting hooks

Environment in parse_config is being used to ensure that settings in the
current run_mode's environment are checked for hooks, and to handle
hooks that are simply on_write.  In this latter case, the code looks up
the value of the setting from all available elements of the setting's
search_path including the current environment layer.

But if directory environments are in use, their environment settings
cannot be looked up at this early point in settings initialization (we
have no environment loaders), and the consequence is that we end up with
a ChainedValues instance cached in the Settings @values[:env][:runmode]
that does not have a section for :env.  This then blocks resolution of
the real environment settings at a later date.

The fix in this patch is to construct a temporary ChainedValues instance
for the lookup rather than invoking value(), thus avoiding the cacheing.

Since none of the documented environment settings has a hook, we were
very tempted to yank out the environment handling entirely, but this
would leave us with yet more odd behavior if a hook was added to an
environment setting in the 3.7.x series for some unknowable reason, and
it was then set in a legacy environment block.

* the question of how to handle Setting hooks for directory environment
settings is open in PUP-3507

Paired-with: Britt Gresham <britt@puppetlabs.com>
Merge pull request #3228 from jpartlow/issue/stable/pup-3500-avoid-cacheing-chained-values-when-handling-setting-hooks

(PUP-3500) Avoid cacheing environment values when handling setting hooks
(PUP-3370) Remove serialization support from indirector request

Support for serializing/deserializing indirector requests was added per
Redmine issue #7080, but was never fully fleshed out and is a rather
sketchy idea in the first place. This commit removes all serialization
logic from Puppet::Indirector::Request.
(PUP-3277) Use classifier when testing enc response

The current test sets up an enc instance for foss, but when running
against pe, it fails because the pe node_terminus configuration is in
[master] not [main].  We could adjust that, but then we
are not testing against a real PE environment, so instead, use the
classifier.
Merge branch 'issue/master/pup-3277-enc-non-existent-directory-environment-test-uses-classifier-in-pe'

* issue/master/pup-3277-enc-non-existent-directory-environment-test-uses-classifier-in-pe:
  (PUP-3277) Use classifier when testing enc response
(PUP-3300) Remove extend self from Checksums

This commit modifies Puppet::Util::Checksums to not extend self anymore
to move to using module_function.
(PUP-3289) Remove deprecated catalog_format and dynamicfacts settings

This commit is part of the Puppet 4 code removal effort.
(PUP-2609) Make copying permissions on Win an error

Copying permissions on Windows are problematic, so it isn't done. Change
the deprecation warning about source_permissions => use/use_when_creating
to an error.
Merge remote-tracking branch 'upstream/pr/3230'

* upstream/pr/3230:
  (PUP-3370) Remove serialization support from indirector request
(PUP-3130) Remove hidden _timestamp fact

Puppet generates a timestamp when a Puppet::Node::Facts object is
created. During PSON serialization (from agent to master), the timestamp
is serialized separately from the hash of fact values.

Previously, when the master deserialized the facts from JSON, it would
convert the timestamp to a hidden fact named `_timestamp`, inline with
the normal facts. As a result, there was a `_timestamp` variable
available in top-scope.

The `_timestamp` fact was added in 5aa4440b circa 2007. It was
originally called `_puppet_timestamp`, and then renamed to `_timestamp`
in a6fe7005.

This commit modifies the Fact object so that it no longer adds a
`_timestamp` fact during deserialization. Note the timestamp is still
available, via the Facts#timestamp method, it's just not a fact.

This commit also removes the magic behavior of stripping facts with
leading underscores. This improves performance because we no longer
duplicate the values hash, or apply a regex to every fact key.

It also preserves behavior of the compiler, added in fb5f859cf4, which
updates the facts timestamp prior to saving them.
(PUP-3462) Add the key 'calling_class_path' which is 'calling_class' with '::' by '/'

This enables to have the class hiera yaml files in directories and enables to
have class based hiera data on windows which does not allow ":" in path names.
(PUP-3514) Fix broken errors from future parser

When the future parser called the error method from within the parser
(the few remaining cases handled by the parser directly), that resulted
in the Factory being called to produce new expressions and these were
subsequently turned into strings - showing a series of (slice ...)
s-expression.

The error method was simply doing the wrong thing (copied from the 3x
parser). This commit fixes the problem by picking up as much as possible
of the known location and passing that along in the exception that is
raised.

One test is included to ensure we do not have regressions on this
behavior. The remaining error cases are still untested, but the
intention is for those to be moved out from the grammar and into
validation to improve the ability to give better error messages.

This commit only fixes the immediate problem of getting garbage output
instead of file, line and position in these cases.
Merge pull request #3229 from hlindberg/PUP-3505_given-undef-not-considered-missing

(PUP-3505) Fix problem with passing explicit undef res parameter