(#10221) Update puppetd plugin to take advantage of new agent statuses

Ticket #9923 allowed the puppetd plugin to trigger agent runs even when
the agent was already daemonized, but it introduced some new state
information about the puppet agent, :idling and :stopped, that wasn't
consolidated very well with the current understanding of puppet state by
other areas on the plugin.

This commit consolidates that state information to reply[:status], and
adds the reply[:idling] and reply[:stopped], like the reply[:running]
and reply[:enabled], for backward compatibility and easy command line
manipulation of the status information.

The agent, application and ddl were all updated to take advantage of
this new state information.  Also bumped the version from 1.4 to 1.5.
Merge pull request #31 from mmrobins/10221_puppetd_status_rebased

(#10221) Update puppetd plugin to take advantage of new agent statuses
11941 - mco nettest - not working

There was a validation error in the DDL file that prevented the
application from working correctly - output was not shown etc.

Additionally cleared up the spacing inconsistencies in the agent
and forced the port argument to be an Integer
11941 - mco nettest - not working

Refactor the application to be easier to debug and test, add
tests for the application and agent
Don't die in an unclear way if you forget the registration is Meta config
Allow inserting extra key into registration data.

From a directory of arbitrary YAML files which I want to get pushed up into Mongo with registration.

11:10 < t0m> Volcane: What's the reason behind squashing facts.yaml to be 1 level deep? I was considering doing something devious by making facts.yaml more complex (and using that data from mongodb), but it's not going to fly as facts/yaml.rb explicitly squashes the facts..
11:11 < oholiab> t0m: are you writing to facts.yaml from facter?
11:12 < t0m> I was writing to facts.yaml with vi to work out if it worked :)
11:18 < oholiab> so are you effectively trying to nest hashes and it won't work?
11:41 -!- e1mer [~erivera@unaffiliated/e1mer] has quit [Ping timeout: 255 seconds]
12:20 < Volcane> t0m: was all i needed initially for facter compat but been toying with ways to do nested data sanely - problem is mostly how do you use nested data on the cli
12:24 < t0m> Volcane: nod. I don't care about that so much though :) I have just smashed it, and I'm getting the deeper data into mongodb (https://github.com/bobtfish/marionette-collective/commit/a30941fd512ae9563c7e22a702996b0024b94501)
12:24 < Volcane> t0m: though been playing a lot with all that recently since the new rich filter language is effectively working on nested data and I am also supporting outputing JSON now and we wrote a grep tool for that that can grep in nested data
12:25 < t0m> cool, I haven't kept up with any of the new stuff, I'm still mostly playing catch up to mongodb stuff.. ;)
12:25 < Volcane> mco rpc puppetd last_run_summary -j|jgrep "data.resources.failed>0" -s sender
12:25 < Volcane> finds all machines that have failing resources, prints their mcollective id
12:26 < t0m> my actual problem which prompted this is I want to generate a load of graphs dynamically based on what's on the FS... I generate a yaml file with the data on the host, but I need to know a list of graphs to gen on the puppetmaster basically..
12:26 < Volcane> t0m: ah
12:26 < t0m> generate('/bin/cat', '/ohshit/this/is/on/wrong/system.yaml')
12:26 < Volcane> hehe
12:27 < t0m> etc etc :)
12:27 < Volcane> that jgrep did a grep over documents like http://www.pastie.org/2247985 one per host
12:27 < Volcane> so i am getting fairly happy with nested data on the cli, so might one day support it in facts
12:27  * t0m nods. That's exactly what I want really..
12:28 < t0m> Would be to make the registration plugin be able to pull in extra .yaml files, just for registration
12:28 < t0m> rather than crapping into facts, which is basically, a hack :)
12:28 < Volcane> yeah, i do that in mine
12:29 < t0m> ah, code visible anywhere?
12:29 < Volcane> also send registration data to a queue rather than to another mc agent cos i do entirely different things with reg
12:29 < t0m> cause that's what I really want, I'm just being a cheap-ass :)
12:29 < Volcane> nah but it should be trivial - look at the yaml.rb it already support merging yamls
12:29 < Volcane> so just nick that code
12:31 < t0m> I'll probably do that actually, if only for the fact I can then subclass your code, rather than maintaining a patch on it. Unless you can be persuaded to remove the flattening from the facts :)
12:32 < Volcane> well you wont see such a change soon is all i can say:)
12:32 < Volcane> we're on like a 6ish month production release cycle
12:33 < t0m> sure, I don't need to see it in a real release :)
12:33 < Volcane> https://github.com/puppetlabs/marionette-collective/blob/master/plugins/mcollective/facts/yaml_facts.rb#L23-33
12:33 < Volcane> thats how i merge the yamls in the fact source
12:33 < t0m> I'm happy for $work to work off master.. I'm just less happy about having to maintain own rebased branch.
12:34 < t0m> yy, I've spent the last couple of hours working out how all of this works ;)
12:34 < t0m> so I guess I'm looking for some advice for what patch would be acceptable upsteam (even if things in a release don't change for 6 months) to enable me to get custom things into the registration data :)
12:35 < t0m> *upstream
12:35 < Volcane> just write your own registration plugin - thats why its a plugin, the expectation is that most people will have their own
12:35 < t0m> this implies that I'm maintaining ruby :)
12:36 < Volcane> like they're this easy https://github.com/puppetlabs/marionette-collective/blob/master/plugins/mcollective/registration/agentlist.rb
12:36 < t0m> I'm entirely happy to write ruby. I just don't want to maintain it :)
12:36 < t0m> sure.
12:36 < t0m> I was hoping to be able to make the general one a bit more flexible, and then never have to touch it again, as maintaining it wouldn't be my problem ;)
12:37 < Volcane> ok, so https://github.com/puppetlabs/mcollective-plugins/blob/master/registration/meta.rb and around line 27 slap in the merge bit
12:37 < Volcane> and that I'd happily accepot
12:37 < Volcane> accept
12:39  * t0m nods - ok, cool, that's what I was after. So, just so that we're crystal, I'm going to add another hash to registration data which gets filled with yamls from config, or add the ability to merge extra .yamls onto the facts here?
12:39 < Volcane> hmm, probably best to do into a new key
12:39 < t0m> that was my thought also.
12:40 < Volcane> and make that key so it could be multiple yaml files merged up together
12:40 < t0m> I'm gonna go play with that then, I shall bug & pull request when I have something sane(r)
12:40 < t0m> thanks!
12:40 < Volcane> kewl kewl

This was a while ago - so if this is no longer appropriate or you have better ideas, please let me know
and I'll be happy to re-work.
12503 - issue when installing packages (puppet 2.7.x, mcollective 1.2.1, package.rb 2.1)

On debian/ubuntu absent packages are reported as :purged by Puppet,
this is a recent change in either the distro behavior or Puppet.

This commit adjusts the agent and tests to consider both :absent and
:purged to indicate a package being absent for logic in install, update
and uninstall actions.
(#7040) Provide an application for wrapping dsh using mcollective discovery
Merge pull request #35 from kbarber/ticket/master/7040-dsh_app

(#7040) Provide an application for wrapping dsh using mcollective discovery
7406 - Turn mc-pgrep into an application

Turn the basic mc-pgrep into an mco application.  As with mc-pgrep the
kill and pkill actions are not represented in the application as they
are just too dangerous for general use.

The timeout in the agent did not match up with the DDL, this resolves
that avoiding short timeout errors on slow machines

The mc-pgrep script was removed
Merge pull request #34 from bobtfish/registration_topic_extra_yamls_in_registration_data

Registration topic extra yamls in registration data
12503 - issue when installing packages (puppet 2.7.x, mcollective 1.2.1, package.rb 2.1)

Update the DDL to match version in the package agent
The check could fail in cases where a node sent completely bogus results
we now handle that as unknown data.

The client never disconnected from the middleware resulting in lots of
EOF exceptions being logged in ActiveMQ

Fixed the indention
14543 - Report versions in the last_run_summary action

Recent versions of puppet has added more information to the
last_run_summary file so we're reproting that now too
13412 - puppetd plugin should use "puppet agent" and not puppetd

Change the default puppetd command from /usr/sbin/puppetd to
/usr/bin/puppet agent.  The puppetd command has been deprecated for
sometime and upcoming puppet 3.0 will completely remove it.
7760 - sshkey exception handling and logging

The sshkey plugin is unused, it does not work with mcollective 2.0 and
it is not currently somehting we want to maintain as part of the core
project or set of plugins.

Removing it from this repository.
13287 - Separate service and package puppet code into util classes

In the past the service and package agent have been totally reliant on
puppet being installed.

The puppet implementation have been seperated from the service and
package agents. The puppet-package and puppet-service agents have
been renamed to package and service respectively.

The actual service and package implementations are now found in
util/puppetpackage.rb and util/puppetservice.rb and can be interchanged
with other implementations. The one that is going to be used in specified
in server.cfg for example

plugin.package.provider = puppet

  and

plugin.service.provider = puppet