I don't think we should be applying this check. If the content has "binaryEncoding" then we should represent it as a byte array, it's up to the user to configure their binding library in whatever way is required to support the relevant binary encoding. Gson (for instance) requires a custom serializer/deserializer to be set - there's no reason that this custom serializer/deserializer has to use base64.

Do you require this strict check for your use-case?