(#15791) isolate deterministic random function

We should not fiddle with the global seed, as other people
might still depend on Kernel.rand() not being that deterministic
as it is the idea with the fqdn_rand function.

However, as ruby < 1.9.2 does not provide such a nice solution, as
a dedicated Random class we simply call Kernel.srand() again on ruby
< 1.9.2 -> Reinitialize with a less deterministic seed.
(#11331) Add pops - this is the base implementation of new DSL model.

This adds a pops basic implementation of the DSL model, an evaluator,
the start of a validator, infrastructure classes and tests.

This commit contains the first working version of parsing. New test
parses a simple puppet DSL expressions and returns the expected result.

This adds proper handling of numbers:
- an lex error if a number is not formatted correctly
- parser creates QualifiedName or LiteralNumber for NAME token
- tree dumper outputs numbers with specified radix

Add support for match expr with quoted text as RHS.

This adds support for expressions like 'a'=~ "ends with ${foo}" to
interpolate into the regular expression (a string is compiled into a
regular expression).

Note that lexer now does not produce a REGEXP after INHERITS as it used
to in earlier versions. (3.1 flags inherits with regexp as syntax error
even if it is allowed in the grammar).

Add parse test for some 3.x "weird" class and define syntax.

This includes performing look ahead in the lexer (for {|) and emitting a
LAMBDA token instead of a LBRACE for the lambda start LBRACE.
(#11331) Add support for foreach.

This adds support for "foreach" for arrays and hashes.
The implementation adds support for MethodCall and Lambda expressions to
the lexer and parser.

A method call is simply a variant of function call that supports an
optional parameterized block (i.e. a lambda). The foreach method is
simply implemented as a function that accepts a "receiver" and a lambda.
The foreach function operates on arrays and hashes.

The implementation does not alter any existing syntax and does not
introduce any additional keywords or constraints on any lexer literals.

Since racc only has one token lookahead it was necessary to add such
lookahead in the lexer for LBRACE, which now produces a LAMBDA token if
 LBRACE is followed by PIPE. DOT token was reintroduced, and a PIPE
token added.

To allow a lambda to return the result of a single expression, the
syntax: {|...| = expr } may be used. Unfortunately, a major rewrite of
the grammar is required to allow this without a syntactic marker. Also
unfortunate is the fact that all functions invokable as statements does
not produce an rvalue and an rvalue function may not be used as a
statement. (To change that would also be a major rewrite). Hence, a
syntactic marker (in a lambda only), that allows a single expression to
be turned into a "rvalue" statment seems to be the best approach). If
the grammar is improved in the future, the = could be made optional.
(#11331) Add support to end a lambda with = expression.

This relaxes the constraint that a = expr must be the only statement
in a lambda (or be a sequence of statements). Is is now instead possible
to place this statement last after a sequence of statements.

This commit also fixes an issue where the last value of the lambda and
foreach function did not produce a value.
(#11331) Make implementation of lambda call work under Ruby 1.8.7

Under some conditions, array zip in ruby 1.8.7 produces a single value
instead of an array. The result was that the integer value of the first
char was produced as the result instead of the expected string.

This fix checks if the name value is just a string name, or an array
with [name, value].
(yardoc) Correct yardoc for code related to #11331 add foreach.
(#18764) Add methods for enumerable functions.

This adds support for the functions/methods collect, reject, select, and
reduce. This implementation depends on support for lambdas being added
in puppet as described in issue #11331). Which probably will be released
in puppet 3.2.
(#11331) Add support for foreach.

This adds support for "foreach" for arrays and hashes.
The implementation adds support for MethodCall and Lambda expressions to
the lexer and parser.

A method call is simply a variant of function call that supports an
optional parameterized block (i.e. a lambda). The foreach method is
simply implemented as a function that accepts a "receiver" and a lambda.
The foreach function operates on arrays and hashes.

The implementation does not alter any existing syntax and does not
introduce any additional keywords or constraints on any lexer literals.

Since racc only has one token lookahead it was necessary to add such
lookahead in the lexer for LBRACE, which now produces a LAMBDA token if
 LBRACE is followed by PIPE. DOT token was reintroduced, and a PIPE
token added.

To allow a lambda to return the result of a single expression, the
syntax: {|...| = expr } may be used. Unfortunately, a major rewrite of
the grammar is required to allow this without a syntactic marker. Also
unfortunate is the fact that all functions invokable as statements does
not produce an rvalue and an rvalue function may not be used as a
statement. (To change that would also be a major rewrite). Hence, a
syntactic marker (in a lambda only), that allows a single expression to
be turned into a "rvalue" statment seems to be the best approach). If
the grammar is improved in the future, the = could be made optional.
(#11331) Add support to end a lambda with = expression.

This relaxes the constraint that a = expr must be the only statement
in a lambda (or be a sequence of statements). Is is now instead possible
to place this statement last after a sequence of statements.

This commit also fixes an issue where the last value of the lambda and
foreach function did not produce a value.

There was an inspect in a rescue that was only there for debugging.
Now removed.
(#11331) Make implementation of lambda call work under Ruby 1.8.7

Under some conditions, array zip in ruby 1.8.7 produces a single value
instead of an array. The result was that the integer value of the first
char was produced as the result instead of the expected string.

This fix checks if the name value is just a string name, or an array
with [name, value].
(yardoc) Correct yardoc for code related to #11331 add foreach.
(#11331) Make lambdas optionally available on all function calls.

This makes it possible to invoke foreach as:
foreach($a) {|$x| ... }

The only caveat is that a parentheses-less call must have at least one
argument. This is not allowed:
foo {|$x| }

If someone implements a function that takes only a lambda that call
would need to be written as:
foo() {|$x| }

This since it is not allowed to pass a lambda as an argument (since this
would require implementing real closures).
(#11331) Add expression based grammar. Switch tests to use egrammar.

This completes the eparser grammar (expression parser). All tests except
one is passing (expected since it checks for an parse error for logic
that is now acceptable; needs some test refactoring as it should apply
only to the regular grammar).
(#11331) Add support for both lambda param inside and outside braces.

This supports {|$x| ... } as well as |$x| { ... }.

The dual support is to allow for convenient UX investigation.
(#11331) Add transformation from pops model to AST

This transformer is intended to be used after parsing to a pops
model. It transforms to a 3.1 AST. The intent is to ensure the exact
same behavior when evaluating. (This is a step on a longer migration
path).

Add Factory#respond_to? since method_missing is implemented.

A method_missing implementation should always also implement
respond_to?

(maint) Fix comment stating that [] is "probably never used".
It is certainly used for testing.

Add AST::BlockExpression and use this instead of ASTArray for statements

An AST::BlockExpression is now used instead of ASTArray to represent
a block of expressions/sequence of expressions when the intent is not
to collect the result of each expression/statement (as ASTArray does).

The grammar is changed to reflect this, and so is merging of classes
that now compose the result instead of appending (modifying).

Name classes with full name when parsing (like 3.1).

It is required that classes are fully named when parsed (i.e.
if nested, they must have the fully expanded name set). This
because they may be lazily loaded. This is required in order to
be able to transform a pops model to 3.1 AST.

(w.i.p) transformer is now wired for lambdas.

This add transformation of pops model to AST for lambda related
expressions. (This is w.i.p, there are no tests yet).
(#18764) This adds 'each' function (a copy of foreach).

This is done to enable Ux testing of both styles.
(Only one of them should remain in a released version).
(#11331) Refactor ModelTreeDumper to abstract TreeDumper and specialization.

This is to prepare for adding an AST tree dumper.
(#11331) Improve model to AST transformation - add tests

Add AstTreeDumper.
Add transformation test for basic expressions.
Add transform conditionals test.
Add tests for transformed calls.
Adds tests for function and method calls transformed to AST.
Add transformer containers tests (class, define, node).

Add test for resource override of multiple resources.
This test was missing (e.g. for File['x', 'y'] ...)

Add test for transformation of resource expression and collection.

Add test for rgen sanity_check regarding Hash[[]].

RGen must either have been fixed or monkey patched so that
its extension of Array (method_missing for an empty array) does not
produce an Array (when it should have passed the task to super).

Remove pending status from test now that egrammar is in use.

It was not possible to parse [1,2,3][2] prior to egrammar.

Cleanup AstTreeDumper, AstTransformer

Removes cruft from copy of ModelTreeDumper, and fixes trailing spaces
on empty lines.
(maint) Update egrammar lexer to latest standard.

This merges changes from master's lexer version using
new mechanism for checking if token is acceptable in context.
(maint) Cleanup empty lines, unused code and comments in pops model.