(PE-6014) Create NOOP CA service

Is this basically the same try/catch logic as the `with-puppet-conf` testutil?  If so, do we want to just figure out a way to re-use that instead of copypasta?