BSON serializing error should be marked with INVALID_BSON

RUBY-676