Add tests for setCustomOption().

Includes a small fix for the heap1 storage engine since
ErrorCodes::InvalidOptions should be returned for a unsupported
option instead of ErrorCodes::BadValue.