session.compact calls checkpoint, which can't be called inside a
transaction; give session.compact its own test and error message so
application writers aren't confused.
Add simple transaction support to test/format.
session_api.c: function with return after a jump to an error label