set the order#payment_state to the correct state when order is canceled

Fixes #5278
Closes #5279

return the new payment state in the update_payment_state, and add some spacing in the spec.

remove the stub to make sure this is not always returning "pending" *doh*