Fix checkout notify callback docs and tests

The checkout notify callback behavior on non-zero return values
was not being tested.  This adds tests, fixes a bug with positive
values, and clarifies the documentation to make it clear that the
checkout can be canceled via this mechanism.