Let's call "node 1" the one that has the keystone vip node and "node 2" the node that does not.
Here is a paste from node 2 puppet output without the exec restart grep'ing for rsync|xinetd|the vip:
http://fpaste.org/91725/62917413/
Note line 56 shows the rsync failure.
Here is a paste from node 2 puppet output with the exec restart:
http://fpaste.org/91727/96629233/
The rsync succeeds and xinetd restarted manually as expected.

Both of these are from a fresh install.  Note the ordering of things is the same on node 1, the point being that xinetd will have been restarted by the time node 2 or node 3 comes around and tries to rsync in the latter case.

If I tried to manually rsync to the primary node after the puppet run in the no-exec-restart case, it also failed with "connection refused".

There is the irritating observation that the rsync stuff just happens before the vip is up, despite a puppet dependency from quickstack::pacemaker::vips["$keystone_group"] to quickstack::pacemaker::vips["$keystone_group"] .  So, not sure why that isn't working as expected.  Or why "~> service['xined']" doesn't restart xinetd.  But for now, this xinetd restart works.