Handle YAML load errors from the new Psych YAML parser.

Earlier versions of Ruby used the syck library to parse YAML; during 1.9 this
moved to the Psych parser based on libyaml.  This brings YAML 1.1 compliance
to the library, and generally moves in a forward direction.

It also makes some radical changes to the failure modes, since Psych is much
more forgiving of some YAML constructs that would previously block the heck
up.

It also, in Ruby tradition, does absolutely nothing to abstract over the
implementation: Ruby can use either Psych or Syck and will raise different
exceptions on failure in both cases.

Worse, Psych exceptions are not derived from StandardError, so
Psych::SyntaxError will actually bypass almost any normal attempt to trap the
error, and will have wildly different details to the Syck errors.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Action definition requires at least one argument.

Ruby 1.9 allows us to enforce the calling convention of face actions at load
time, rather than failing with a runtime error.  This, in turn, means that we
should define actions correctly in our tests, if we expect them to work.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Hash references on string now raise errors, not return nil.

Under Ruby 1.8 the syntax `"foo"[:bar]` would return `nil`, while Ruby 1.9 is
strict: it enforces that only numeric index arguments are acceptable and
raises an error when that precept is violated.

This means that a whole pile of specs that used to pass strings where
structured objects were accepted now blow up - because you can't request
arbitrary nonsense against their array dereference interface.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Eliminate dead home-made assertion-at-dev-time-only code.

The assertion code hidden in here, designed to allow development time
assertions than vanish at runtime - in the classic "better to keep working on
corrupt data and failed assumptions than fail in front of the user" mode - was
not actually used anywhere in the product.

We may as well not carry this little legacy around any further than it current
has been, then.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
`instance_variables` changes return type between 1.8 and 1.9

The return type of `instance_variables` changes between Ruby 1.8 and 1.9
releases; it used to return an array of strings in the form "@foo", but
now returns an array of symbols in the form :@foo.

Nothing else in the stack cares which form you get - you can pass the
string or symbol to things like `instance_variable_set` and they will work
transparently.

Having the same form in all releases of Puppet is a win, though, so we
pick a unification and enforce than on all releases.  That way developers
who do set math on them (eg: for YAML rendering) don't have to handle the
distinction themselves.

In the sane tradition, we bring older releases into conformance with newer
releases, so we return symbols rather than strings, to be more like the
future versions of Ruby are.

We also carefully support reloading, by only wrapping when we don't
already have the original version of the method aliased away somewhere.

This includes a second monkey-patch, on the `resolv` library that ships with
Ruby 1.8.  This is the only bit of code in the core stack that made
assumptions about the return format of names from `instance_methods`.

Specifically, it assumed they were strings, and could be evaluated using
`instance_eval` to fetch out the value.  The fix is to replace that with a
call to `instance_variable_get` which, like it does in Ruby 1.9, avoids the
cost of eval parsing as well as being more robust overall.

I did audit the rest of the standard library on 1.8.5 and 1.8.7 to verify
this.  We only need worry if we start using the SOAP or Tk support.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Syck specific YAML parser tests can only run with the old engine.

Ruby 1.9 moved to using the Psych engine as the default YAML engine, replacing
the older Syck based YAML engine.  They did this without any regard to parse
tree compatibility, or abstraction over the underlying tool.

Consequently, the test that gets very intimate with the encoding format for
the parsed YAML tree is unable to function with the modern engine - it tries
to interact with a pile of methods that just don't match.

This commit disables that test unless the Syck parser is in use.  Really, the
entire test should be an external, abstracted acceptance level test.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Calling map on string now fails, rather than yielding the string.

In Ruby 1.8 `"foo".map` would yield "foo" once, while in Ruby 1.9 it is not
defined as a method.

Various parts of our testing code stub other methods and return strings where
arrays of strings are appropriate; this historically worked because of that
1.8 shim, and no longer does.

The same is true of `String#each`, which fails in the same way.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
`String#to_a` doesn't work in Ruby 1.9

In Ruby 1.8, counterintuitively, `"foo".to_a` would return `["foo"]` - an
array with a single entry, the string.  This went away in Ruby 1.9, likely
as a transition path to a much less confusing set of semantics.

Either way we should stop relying on that behaviour and get with the program.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Regular expression matching on symbols changed in Ruby 1.9

In Ruby 1.8 a regular expression match on a symbol would never match, while in
Ruby 1.9 it is equivalent t a regular expression match on the equivalent
string.  This is very sensible behaviour, but trips up parts of code that
expect an implicit "... but not symbol" match.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Whitespace and formatting cleanup on shellquote spec.

The formatting in the spec file was very odd, with extraneous and random
whitespace around arguments, odd line breaks, and so forth.

It also used a whole pile of temporary variables that it didn't need to.

This cleans that up, without making any functional changes to the code.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Fix the shellquote parser function on Ruby 1.9

The semantics of `String#include?` changed a little in Ruby 1.9, so that you
can't just pass an arbitrary integer and have it check for character
containment.  (...or, perhaps, the semantics of characters vs integers
changed.  Either way, the effect is the same.)

This fixes the code to do the right thing on all platforms; we want to process
the string byte-wise, since that is how sh may well regard it, and we have to
work around some limitations on 1.8.5 semantics of string iterators.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Protect all existing methods when monkey-patching.

Most, but not all, the method we add to objects while monkey-patching were
correctly protected against overriding a core method.  This adds the right
protections to the handful that were missing them.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Handle the real semantics of `String#lines`...

The previous monkey-patching of `String#lines` and `IO#lines` led to their
behaving differently in our code than in standard Ruby code.  This is
generally surprising, and led to some subtle misuses of the method.

This corrects that, by adapting all the abusers to real Ruby calling
conventions over the enumerator interface.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
The return type of `methods` changed in 1.9

This test depended on the return type of `methods`, which went from array of
strings to array of symbols in Ruby 1.9.  Given that it really wanted to test
if the instance responded to the message, we can just directly test that.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
A correct implementation of IO#binwrite for Ruby 1.8

When we added the `IO#binwrite` method to our collection of backported
features, the specification was misinterpreted and the offset was treated as
"offset in the string we are writing".

Instead, it is an instruction to seek in the output file before writing, which
meant that our semantics were all wrong - and tests broke - when we ran
against the native binwrite on Ruby 1.9.

The new implementation is correct in behaviour on all tested cases, and has a
more extensive test suite exercising all the various parts of the behaviour
that we need.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Fix Cisco device canonicalization regular expression on 1.9

The behaviour of the `\b` Regexp match operator, or rather, the classification
of `/` as an end of word character, changed between Ruby 1.8 and 1.9.

Most uses in our code are safe, and this will lead to equally correct
behaviour, but the Cisco device interface canonicalization code used it where
"end of string before newline" was intended.

Moving to the `\Z` match, which imposes the exactly desired condition,
resolves the problem.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Partially fix UTF-8 and PSON encoding tests for Ruby 1.9

With the introduction of real encoding support in Ruby 1.9, various problems
including incorrectly encoded UTF-8 strings crop up.

By flagging the encoding of the spec file to Ruby, and forcing binary encoding
on strings we compare that contain invalid UTF-8, we can bypass some of those
headaches.

Since PSON is, by design, not compatible with JSON and can do useful things
with arbitrary binary data, this is the most reasonably current choice for how
to handle this troublesome set of data.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Implement YAML 1.1 binary data encoding in ZAML

The YAML spec has a mechanism to flag arbitrary binary data, as opposed to
sequences of Unicode characters in UTF-8.  This implements full support for
that in the ZAML encoder.

This is used on the basis of a string that doesn't contain valid UTF-8.
Another stinking heuristic, but at least one that results in conformant YAML
being emitted.

Pleasantly, the !binary tagged data is correctly ready by Syck as far back as
Ruby 1.8.5, allowing us to confidently ship this data without too much concern
for compatibility with older libraries.

Additionally, we stop escaping all UTF-8 data as a sequence of bytes.
Theoretically we could go back to escaping it, using real Unicode character
markers, but literal UTF-8 in the output file is just as acceptable.

This should gain better support for the Ruby 1.9 encoding markers, so that
strings could carry their binary or non-binary nature in ... but this gets
things working and passes both Unicode and Binary data tests.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
When shutting down Mongrel, wait synchronously for stop.

The Mongrel tests on Ruby 1.9 were highly unreliable, sometimes failing
because they tried to bind a port that was already in use.

This turns out to be because the default shutdown mode is to asynchronously
stop workers - combined with the fact that our test made a TCP connection to
the service, but never actually completed the request.

That led to one thread that dangled, if the socket didn't close in a timely
fashion, and blocked up the listen port.

Instead, two things: first, wait for all threads to terminate (or 60 seconds
to pass) in the stop method, and second, use a full HTTP request to test, not
just a TCP connection with no content.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>
Use PID-based ports to test web server listening.

When running more than one instance of the spec suite concurrently on a single
machine, the hard-coded listening ports in some of the integration tests could
lead to failures - when multiple suites hit the same tests at the same time.

This uses a port number based off the PID, which helps ensure we don't get
spurious failures from concurrency.

Signed-off-by: Daniel Pittman <daniel@puppetlabs.com>