set the order#payment_state to the correct state when order is canceled

Fixes #5278
Closes #5279
Closes #5388

return the new payment state in the update_payment_state, and add some spacing in the spec.

remove the stub to make sure this is not always returning "pending" *doh*

Conflicts:
	core/app/models/spree/order.rb
Fix specs and update syntax.