Right. The error message in pre-prod is in a different order.

       Expected a full match of the result
      
         {"error"=>
         ["{\"non_existent_cookbooks\":[\"this_does_not_exist\"],\"cookbooks_with_no_versions\":[],\"message\":\"Run list contains invalid items: no such cookbook this_does_not_exist.\"}"]}
      
      
       to the spec
      
         {"error"=>
         ["{\"message\":\"Run list contains invalid items: no such cookbook this_does_not_exist.\",\"non_existent_cookbooks\":[\"this_does_not_exist\"],\"cookbooks_with_no_versions\":[]}"]}

But I agree with you, if the error message is parsed by the client, then we need to test that parsing... Any suggestions?