protostuff/protostuff — pull request #39 by mfycheng (2014-10-30 00:19:56+00:00)
Title: Java 8 Encoding/Decoding Fix

## Motivation

Java 8's built in String class no longer accepts 3-byte surrogates / 6-byte surrogate pairs. This causes issues when UTF characters, such as emoji's, are present in a string. Since Protostuff encodes using 3-byte surrogates / 6-byte surrogate pairs, deserialization using Java's String class results in a corrupted String.

The reason why Java 8 no longer supports 3-byte surrogates / 6-byte surrogate pairs is that it has tightened it's implementation to follow the standard more rigorously (see [here] (https://bugs.openjdk.java.net/browse/JDK-7096080) for details). By following the standards more closely, Protostuff should be compatible with more external sources. For example, Protobuff running on Java 8.

## Changes

* Support 3-byte surrogates / 6-byte surrogate pairs in Java 8, which allows for communication with older version of protostuff.
* Encode surrogate pairs using Standard UTF-8, instead of 6-byte surrogate pairs to avoid corruption for decoders without 3-byte surrogate / 6-byte surrogate pair support.

## Effects

This change should be fully compatible with previous versions of Protostuff, as well as Protobuff's Java implementation (and most likely other language implementations as well). The serialized output should be more in line with what standardized UTF-8 decoders expect.

DataInputStream, however, will not the ability to reliably decode Strings that contain 4-byte surrogates pairs, as they have no support for 4-byte encoded characters. There are many ways around this, however, so it is not a huge concern.