Chapter 2. Marshalling custom objects with ProtoStream Marshalling is a process that converts Java objects into a binary format that can be transferred across the network or stored to disk. The reverse process, unmarshalling, transforms data from a binary format back into Java objects. Data Grid performs marshalling and unmarshalling to: Send data to other Data Grid nodes in a cluster. Store data in persistent cache stores. Transmit objects between clients and remote caches. Store objects in native memory outside the JVM heap. Store objects in JVM heap memory when the cache encoding is not application/x-java-object . When storing custom objects in Data Grid caches, you should use Protobuf-based marshalling with the ProtoStream marshaller. 2.1. ProtoStream marshalling Data Grid provides the ProtoStream API so you can marshall Java objects as Protocol Buffers (Protobuf). ProtoStream natively supports many different Java data types, which means you do not need to configure ProtoStream marshalling for those types. For custom or user types, you need to provide some information so that Data Grid can marshall those objects to and from your caches. SerializationContext A repository that contains Protobuf type definitions, loaded from Protobuf schemas ( .proto files), and the accompanying marshallers. SerializationContextInitializer An interface that initializes a SerializationContext . Additional resources org.infinispan.protostream.SerializationContext org.infinispan.protostream.SerializationContextInitializer 2.1.1. ProtoStream types Data Grid uses a ProtoStream library that can handle the following types for keys and values, as well as the unboxed equivalents in the case of primitive types: byte[] Byte String Integer Long Double Float Boolean Short Character java.util.Date java.time.Instant Additional type collections The ProtoStream library includes several adapter classes for common Java types, for example: java.math.BigDecimal java.math.BigInteger java.util.UUID java.util.BitSet Data Grid provides all adapter classes for some common JDK classes in the protostream-types artifact, which is included in the infinispan-core and infinispan-client-hotrod dependencies. You do not need any configuration to store adapter classes as keys or values. However, if you want to use adapter classes as marshallable fields in ProtoStream-annotated POJOs, you can do so in the following ways: Specify the CommonTypesSchema and CommonContainerTypesSchema classes with the dependsOn element of the ProtoSchema annotation. @ProtoSchema(dependsOn = {org.infinispan.protostream.types.java.CommonTypes, org.infinispan.protostream.types.java.CommonContainerTypes}, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "example") public interface LibraryInitalizer extends SerializationContextInitializer { } Specify the required adapter classes with the includeClasses element of the ProtoSchema annotation @ProtoSchema(includeClasses = { Author.class, Book.class, UUIDAdapter.class, java.math.BigInteger }, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "library") public interface LibraryInitalizer extends SerializationContextInitializer { } Additional resources Protocol Buffers Data Grid ProtoStream API 2.1.2. ProtoStream annotations The ProtoStream API includes annotations that you can add to Java applications to define Protobuf schemas, which provide a structured format for your objects. This topic provides additional details about ProtoStream annotations. You should refer to the documentation in the org.infinispan.protostream.annotations package for complete information. Proto @Proto defines a Protocol Buffers message without the requirement of having to annotate all fields with the @ProtoField annotation. Use this annotation to quickly generate messages from records or classes with public fields. Fields must be public and they will be assigned incremental numbers based on the declaration order. It is possible to override the automated defaults for a field by using the ProtoField annotation. Warning Use automatic Protobuf field numbering only for quick prototyping. For production environments you should follow the Protocol Buffers best practices in order to guarantee future/backwards compatibility with your schema. ProtoField @ProtoField defines a Protobuf message field. This annotation applies to fields as well as getter and setter methods. Unless you are using the @Proto annotation, a class must have at least one field annotated with @ProtoField before Data Grid can marshall it as Protobuf. Parameter Value Optional or required Description number Integer Required Tag numbers must be unique within the class. type Type Optional Declares the Protobuf type of the field. If you do not specify a type, it is inferred from the Java property. You can use the @ProtoField(type) element to change the Protobuf type, similarly to changing Java int to fixed32 . Any incompatible declarations for the Java property cause compiler errors. collectionImplementation Class Optional Indicates the actual collection type if the property type is an interface or abstract class. javaType Class Optional Indicates the actual Java type if the property type is an abstract class or interface. The value must be an instantiable Java type assignable to the property type. If you declare a type with the javaType parameter, then all user code must adhere to that type. The generated marshaller for the entry uses that implementation if it is unmarshalled. If the local client uses a different implementation than declared it causes ClassCastExceptions. name String Optional Specifies a name for the Protobuf schema. defaultValue String Optional Specifies the default value for fields if they are not available when reading from the cache. The value must follow the correct syntax for the Java field type. ProtoFactory @ProtoFactory marks a single constructor or static factory method for creating instances of the message class. You can use this annotation to support immutable message classes. All fields annotated with @ProtoField must be included in the parameters. Field names and parameters of the @ProtoFactory constructor or method must match the corresponding Protobuf message, however, the order is not important. If you do not add a @ProtoFactory annotated constructor to a class, that class must have a default no-argument constructor, otherwise errors occur during compilation. ProtoSchema @ProtoSchema generates an implementation of a class or interface that extends SerializationContextInitializer . If active, the ProtoStream processor generates the implementation at compile time in the same package with the Impl suffix or a name that you specify with the className parameter. The includeClasses or basePackages parameters reference classes that the ProtoStream processor should scan and include in the Protobuf schema and marshaller. If you do not set either of these parameters, the ProtoStream processor scans the entire source path, which can lead to unexpected results and is not recommended. You can also use the excludeClasses parameter with the basePackages parameter to exclude classes. The schemaFileName and schemaPackageName parameters register the generated Protobuf schema under this name. If you do not set these parameters, the annotated simple class name is used with the unnamed, or default, package. Schema names must end with the .proto file extension. You can also use the marshallersOnly to generate marshallers only and suppress the Protobuf schema generation. The ProtoStream process automatically generates META-INF/services service metadata files, which you can use so that Data Grid Server automatically picks up the JAR to register the Protobuf schema. The dependsOn parameter lists annotated classes that implement SerializedContextInitializer to execute first. If the class does not implement SerializedContextInitializer or is not annotated with ProtoSchema , a compile time error occurs. ProtoAdapter @ProtoAdapter is a marshalling adapter for a class or enum that you cannot annotate directly. If you use this annotation for: Classes, the annotated class must have one @ProtoFactory annotated factory method for the marshalled class and annotated accessor methods for each field. These methods can be instance or static methods and their first argument must be the marshalled class. Enums, an identically named enum value must exist in the target enum. ProtoName @ProtoName is an optional annotation that specifies the Protobuf message or enum type name. It can be used on classes, records and enums. ProtoEnumValue @ProtoEnumValue defines a Protobuf enum value. You can apply this annotation to members of a Java enum only. ProtoReserved and ProtoReservedStatements @ProtoReserved and @ProtoReservedStatements add reserved statements to generated messages or enum definitions to prevent future usage of numbers, ranges, and names. ProtoTypeId @ProtoTypeId optionally specifies a globally unique numeric type identifier for a Protobuf message or enum type. Note You should not add this annotation to classes because Data Grid uses it internally and identifiers can change without notice. ProtoUnknownFieldSet @ProtoUnknownFieldSet optionally indicates the field, or JavaBean property of type {@link org.infinispan.protostream.UnknownFieldSet} , which stores any unknown fields. Note Data Grid does not recommend using this annotation because it is no longer supported by Google and is likely to be removed in future. Other annotations Data Grid copies any other annotations on classes, fields, and methods as comments in the generated Protobuf schema. This includes indexing annotations such as @Indexed and @Basic . Additional resources org.infinispan.protostream.annotations Protocol Buffers Language Guide - Reserved Fields Protocol Buffers Language Guide - Reserved Values 2.2. Creating serialization context initializers A serialization context initializer lets you register the following with Data Grid: Protobuf schemas that describe user types. Marshallers that provide serialization and deserialization capabilities. From a high level, you should do the following to create a serialization context initializer: Add ProtoStream annotations to your Java classes. Use the ProtoStream processor that Data Grid provides to compile your SerializationContextInitializer implementation. Note The org.infinispan.protostream.MessageMarshaller interface is deprecated and planned for removal in a future version of ProtoStream. You should ignore any code examples or documentation that show how to use MessageMarshaller until it is completely removed. 2.2.1. Adding the ProtoStream processor Data Grid provides a ProtoStream processor artifact that processes Java annotations in your classes at compile time to generate Protobuf schemas, accompanying marshallers, and a concrete implementation of the SerializationContextInitializer interface. Procedure Add the protostream-processor to the annotation processors configuration of maven-compiler-plugin to your pom.xml . <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>...</version> <configuration> <annotationProcessorPaths> <annotationProcessorPath> <groupId>org.infinispan.protostream</groupId> <artifactId>protostream-processor</artifactId> <version>...</version> </annotationProcessorPath> </annotationProcessorPaths> </configuration> </plugin> </plugins> </build> 2.2.2. Adding ProtoStream annotations to Java classes Declare ProtoStream metadata by adding annotations to a Java class and its members. Data Grid then uses the ProtoStream processor to generate Protobuf schema and related marshallers from those annotations. Procedure Annotate the Java fields that you want to marshall with @ProtoField , either directly on the field or on the getter or setter method. Any non-annotated fields in your Java class are transient. For example, you have a Java class with 15 fields and annotate five of them. The resulting schema contains only those five fields and only those five fields are marshalled when storing a class instance in Data Grid. Use @ProtoFactory to annotate constructors for immutable objects. The annotated constructors must initialize all fields annotated with @ProtoField . Annotate members of any Java enum with @ProtoEnumValue . The following Author.java and Book.java examples show Java classes annotated with @ProtoField and @ProtoFactory : Author.java import org.infinispan.protostream.annotations.ProtoFactory; import org.infinispan.protostream.annotations.ProtoField; public class Author { @ProtoField(1) final String name; @ProtoField(2) final String surname; @ProtoFactory Author(String name, String surname) { this.name = name; this.surname = surname; } // public Getter methods omitted for brevity } Book.java import org.infinispan.protostream.annotations.ProtoFactory; import org.infinispan.protostream.annotations.ProtoField; public class Book { @ProtoField(number = 1) public final UUID id; @ProtoField(number = 2) final String title; @ProtoField(number = 3) final String description; @ProtoField(number = 4, defaultValue = "0") final int publicationYear; @ProtoField(number = 5, collectionImplementation = ArrayList.class) final List<Author> authors; @ProtoField(number = 6) public Language language; @ProtoFactory Book(UUID id, String title, String description, int publicationYear, List<Author> authors, Language language) { this.id = id; this.title = title; this.description = description; this.publicationYear = publicationYear; this.authors = authors; this.language = language; } // public Getter methods not included for brevity } The following Language.java example shows a Java enum annotated with @ProtoEnumValue along with the corresponding Protobuf schema: Language.java import org.infinispan.protostream.annotations.ProtoEnumValue; public enum Language { @ProtoEnumValue(number = 0, name = "EN") ENGLISH, @ProtoEnumValue(number = 1, name = "DE") GERMAN, @ProtoEnumValue(number = 2, name = "IT") ITALIAN, @ProtoEnumValue(number = 3, name = "ES") SPANISH, @ProtoEnumValue(number = 4, name = "FR") FRENCH; } Language.proto enum Language { EN = 0; DE = 1; IT = 2; ES = 3; FR = 4; } Additional resources org.infinispan.protostream.annotations.ProtoField org.infinispan.protostream.annotations.ProtoFactory 2.2.3. Creating ProtoStream adapter classes ProtoStream provides a @ProtoAdapter annotation that you can use to marshall external, third-party Java object classes that you cannot annotate directly. Procedure Create an Adapter class and add the @ProtoAdapter annotation, as in the following example: import java.util.UUID; import org.infinispan.protostream.annotations.ProtoAdapter; import org.infinispan.protostream.annotations.ProtoFactory; import org.infinispan.protostream.annotations.ProtoField; import org.infinispan.protostream.descriptors.Type; /** * Human readable UUID adapter for UUID marshalling */ @ProtoAdapter(UUID.class) public class UUIDAdapter { @ProtoFactory UUID create(String stringUUID) { return UUID.fromString(stringUUID); } @ProtoField(1) String getStringUUID(UUID uuid) { return uuid.toString(); } } Additional resources org.infinispan.protostream.annotations.ProtoAdapter 2.2.4. Generating serialization context initializers After you add the ProtoStream processor and annotate your Java classes, you can add the @ProtoSchema annotation to an interface so that Data Grid generates the Protobuf schema, accompanying marshallers, and a concrete implementation of the SerializationContextInitializer . Note By default, generated implementation names are the annotated class name with an "Impl" suffix. Procedure Define an interface that extends GeneratedSchema or its super interface, SerializationContextInitializer . Note The GeneratedSchema interface includes a method to access the Protobuf schema whereas the SerializationContextInitializer interface supports only registration methods. Annotate the interface with @ProtoSchema . Ensure that includeClasses parameter includes all classes for the generated SerializationContextInitializer implementation. Specify a name for the generated .proto schema with the schemaFileName parameter. Set a path under target/classes where schema files are generated with the schemaFilePath parameter. Specify a package name for the generated .proto schema with the schemaPackageName parameter. The following example shows a GeneratedSchema interface annotated with @ProtoSchema : @ProtoSchema( includeClasses = { Book.class, Author.class, UUIDAdapter.class, Language.class }, schemaFileName = "library.proto", schemaFilePath = "proto/", schemaPackageName = "book_sample") interface LibraryInitializer extends GeneratedSchema { } steps If you use embedded caches, Data Grid automatically registers your SerializationContextInitializer implementation. If you use remote caches, you must register your SerializationContextInitializer implementation with Data Grid Server. Additional resources org.infinispan.protostream.annotations.ProtoSchema 2.2.5. Protocol Buffers best practices The Protocol Buffers documentation provides a list of best practices on how to design messages and how to evolve the schema in order to maintain backwards compatibility. Data Grid can automatically perform compatibility checks when schemas are updated and reject updates when incompatibilities are detected. The types of checks can be configured via the schema-compatibility attribute of the global serialization configuration. The available levels are: UNRESTRICTED : no checks are performed LENIENT : a subset of the rules are enforced STRICT : all the rules are enforced (default) The following table shows the compatibility check rules enabled for each level Rule Description Level No Using Reserved Fields Compares the current and updated definitions and returns a list of warnings if any message's previously reserved fields or IDs are now being used as part of the same message. LENIENT , STRICT No Changing Field IDs Compares the current and updated definitions and returns a list of warnings if any field ID number has been changed. LENIENT , STRICT No Changing Field Types Compares the current and updated definitions and returns a list of warnings if any field type has been changed. LENIENT , STRICT No Removing Fields Without Reserve Compares the current and updated definitions and returns a list of warnings if any field has been removed without a corresponding reservation of that field name or ID. LENIENT , STRICT No Removing Reserved Fields Compares the current and updated definitions and returns a list of warnings if any reserved field has been removed. STRICT No Changing Field Names Compares the current and updated definitions and returns a list of warnings if any message's fields have been renamed. STRICT 2.2.6. Registering serialization context initializers For embedded caches, Data Grid automatically registers serialization contexts and marshallers in your annotated SerializationContextInitializer implementation using the java.util.ServiceLoader . If you prefer, you can disable automatic registration of SerializationContextInitializer implementations and then register them manually. Important If you manually register one SerializationContextInitializer implementation, it disables automatic registration. You must then manually register all other implementations. Procedure Set a value of false for the ProtoSchema.service annotation. @ProtoSchema( includeClasses = SomeClass.class, ... service = false ) Manually register SerializationContextInitializer implementations either programmatically or declaratively, as in the following examples: Declarative <serialization> <context-initializer class="org.infinispan.example.LibraryInitializerImpl"/> <context-initializer class="org.infinispan.example.another.SCIImpl"/> </serialization> Programmatic GlobalConfigurationBuilder builder = new GlobalConfigurationBuilder(); builder.serialization() .addContextInitializers(new LibraryInitializerImpl(), new SCIImpl()); 2.2.7. Registering Protobuf schemas with Data Grid Server Register Protobuf schemas with Data Grid Server to perform Ickle queries or convert from application/x-protostream to other media types such as application/json . Prerequisites Generate Protobuf schema with the ProtoStream processor. You can find generated Protobuf schema in the target/<schemaFilePath>/ directory. Have a user with CREATE permissions. Note Security authorization requires CREATE permissions to add schemas. With the default settings, you need the deployer role at minimum. Procedure Add Protobuf schema to Data Grid Server in one of the following ways: Open the Data Grid Console in any browser, select the Schema tab and then Add Protobuf schema . Use the schema command with the --upload= argument from the Data Grid command line interface (CLI). Include the Protobuf schema in the payload of a POST request with the REST API. Use the generated SerializationContextInitializer implementation with a Hot Rod client to register the Protobuf schema, as in the following example: /** * Register generated Protobuf schema with Data Grid Server. * This requires the RemoteCacheManager to be initialized. * * @param initializer The serialization context initializer for the schema. */ private void registerSchemas(SerializationContextInitializer initializer) { // Store schemas in the '___protobuf_metadata' cache to register them. // Using ProtobufMetadataManagerConstants might require the query dependency. final RemoteCache<String, String> protoMetadataCache = remoteCacheManager.getCache(ProtobufMetadataManagerConstants.PROTOBUF_METADATA_CACHE_NAME); // Add the generated schema to the cache. protoMetadataCache.put(initializer.getProtoFileName(), initializer.getProtoFile()); // Ensure the registered Protobuf schemas do not contain errors. // Throw an exception if errors exist. String errors = protoMetadataCache.get(ProtobufMetadataManagerConstants.ERRORS_KEY_SUFFIX); if (errors != null) { throw new IllegalStateException("Some Protobuf schema files contain errors: " + errors + "\nSchema :\n" + initializer.getProtoFileName()); } } Add a JAR file with the SerializationContextInitializer implementation and custom classes to the USDRHDG_HOME/server/lib directory. When you do this, Data Grid Server registers your Protobuf schema at startup. However, you must add the archive to each server installation because the schema are not saved in the ___protobuf_metadata cache or automatically distributed across the cluster. Note You must do this if you require Data Grid Server to perform any application/x-protostream to application/x-java-object conversions, in which case you must also add any JAR files for your POJOs. steps Register the SerializationContextInitializer with your Hot Rod clients, as in the following example: ConfigurationBuilder remoteBuilder = new ConfigurationBuilder(); remoteBuilder.addServer().host(host).port(Integer.parseInt(port)); // Add your generated SerializationContextInitializer implementation. LibraryInitalizer initializer = new LibraryInitalizerImpl(); remoteBuilder.addContextInitializer(initializer); 2.2.8. Manual serialization context initializer implementations Important Data Grid strongly recommends against manually implementing the SerializationContextInitializer or GeneratedSchema interfaces. It is possible to manually implement SerializationContextInitializer or GeneratedSchema interfaces using ProtobufTagMarshaller and RawProtobufMarshaller annotations. However, manual implementations require a lot of tedious overhead and are prone to error. Implementations that you generate with the protostream-processor artifact are a much more efficient and reliable way to configure ProtoStream marshalling.

Commands:
@ProtoSchema(dependsOn = {org.infinispan.protostream.types.java.CommonTypes, org.infinispan.protostream.types.java.CommonContainerTypes}, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "example") public interface LibraryInitalizer extends SerializationContextInitializer { }
@ProtoSchema(includeClasses = { Author.class, Book.class, UUIDAdapter.class, java.math.BigInteger }, schemaFileName = "library.proto", schemaFilePath = "proto", schemaPackageName = "library") public interface LibraryInitalizer extends SerializationContextInitializer { }
<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>...</version> <configuration> <annotationProcessorPaths> <annotationProcessorPath> <groupId>org.infinispan.protostream</groupId> <artifactId>protostream-processor</artifactId> <version>...</version> </annotationProcessorPath> </annotationProcessorPaths> </configuration> </plugin> </plugins> </build>
import org.infinispan.protostream.annotations.ProtoFactory; import org.infinispan.protostream.annotations.ProtoField; public class Author { @ProtoField(1) final String name; @ProtoField(2) final String surname; @ProtoFactory Author(String name, String surname) { this.name = name; this.surname = surname; } // public Getter methods omitted for brevity }
import org.infinispan.protostream.annotations.ProtoFactory; import org.infinispan.protostream.annotations.ProtoField; public class Book { @ProtoField(number = 1) public final UUID id; @ProtoField(number = 2) final String title; @ProtoField(number = 3) final String description; @ProtoField(number = 4, defaultValue = "0") final int publicationYear; @ProtoField(number = 5, collectionImplementation = ArrayList.class) final List<Author> authors; @ProtoField(number = 6) public Language language; @ProtoFactory Book(UUID id, String title, String description, int publicationYear, List<Author> authors, Language language) { this.id = id; this.title = title; this.description = description; this.publicationYear = publicationYear; this.authors = authors; this.language = language; } // public Getter methods not included for brevity }
import org.infinispan.protostream.annotations.ProtoEnumValue; public enum Language { @ProtoEnumValue(number = 0, name = "EN") ENGLISH, @ProtoEnumValue(number = 1, name = "DE") GERMAN, @ProtoEnumValue(number = 2, name = "IT") ITALIAN, @ProtoEnumValue(number = 3, name = "ES") SPANISH, @ProtoEnumValue(number = 4, name = "FR") FRENCH; }
enum Language { EN = 0; DE = 1; IT = 2; ES = 3; FR = 4; }
import java.util.UUID; import org.infinispan.protostream.annotations.ProtoAdapter; import org.infinispan.protostream.annotations.ProtoFactory; import org.infinispan.protostream.annotations.ProtoField; import org.infinispan.protostream.descriptors.Type; /** * Human readable UUID adapter for UUID marshalling */ @ProtoAdapter(UUID.class) public class UUIDAdapter { @ProtoFactory UUID create(String stringUUID) { return UUID.fromString(stringUUID); } @ProtoField(1) String getStringUUID(UUID uuid) { return uuid.toString(); } }
@ProtoSchema( includeClasses = { Book.class, Author.class, UUIDAdapter.class, Language.class }, schemaFileName = "library.proto", schemaFilePath = "proto/", schemaPackageName = "book_sample") interface LibraryInitializer extends GeneratedSchema { }
@ProtoSchema( includeClasses = SomeClass.class, service = false )
<serialization> <context-initializer class="org.infinispan.example.LibraryInitializerImpl"/> <context-initializer class="org.infinispan.example.another.SCIImpl"/> </serialization>
GlobalConfigurationBuilder builder = new GlobalConfigurationBuilder(); builder.serialization() .addContextInitializers(new LibraryInitializerImpl(), new SCIImpl());
schema --upload=person.proto person
POST/rest/v2/schemas/<schema_name>
/** * Register generated Protobuf schema with Data Grid Server. * This requires the RemoteCacheManager to be initialized. * * @param initializer The serialization context initializer for the schema. */ private void registerSchemas(SerializationContextInitializer initializer) { // Store schemas in the '___protobuf_metadata' cache to register them. // Using ProtobufMetadataManagerConstants might require the query dependency. final RemoteCache<String, String> protoMetadataCache = remoteCacheManager.getCache(ProtobufMetadataManagerConstants.PROTOBUF_METADATA_CACHE_NAME); // Add the generated schema to the cache. protoMetadataCache.put(initializer.getProtoFileName(), initializer.getProtoFile()); // Ensure the registered Protobuf schemas do not contain errors. // Throw an exception if errors exist. String errors = protoMetadataCache.get(ProtobufMetadataManagerConstants.ERRORS_KEY_SUFFIX); if (errors != null) { throw new IllegalStateException("Some Protobuf schema files contain errors: " + errors + "\nSchema :\n" + initializer.getProtoFileName()); } }
ConfigurationBuilder remoteBuilder = new ConfigurationBuilder(); remoteBuilder.addServer().host(host).port(Integer.parseInt(port)); // Add your generated SerializationContextInitializer implementation. LibraryInitalizer initializer = new LibraryInitalizerImpl(); remoteBuilder.addContextInitializer(initializer);