Project: Reactive Interaction Gateway (Orchestration & Management / API Gateway)
File: avro.md

id: avro title: Avro Implementation Details sidebar_label: Avro Apache Avro format as adapted by the : ```bash <0, 0, 0, 0, 1, 5, 3, 8, ...> ``` Adopting Avro for event (de)serialization is fairly straightforward. First you need to run an instance of the `Kafka Schema Registry`, which is a central store for all Avro schemas in use. As an event is consumed from Kafka, RIG fetches the corresponding schema from the registry and deserializes the event with it, caching the schema in the process (in memory). As for producing, RIG again retrieves and caches the schemas used for serializing events. producer evaluates if serialization is turned on by checking `KAFKA_SERIALIZER` environment variable and if it's value is `avro` If it is, creates headers for Kafka event by appending `ce_` prefix for every field, except `data` field - binary mode nested context attributes are stringified, since Kafka headers don't support nested values (this is common when using Cloud events extensions) after that, the `data` field is serialized using the schema name (function for getting schemas from registry is cached in-memory) producer sends event with created headers and data (in binary format `<<0, 0, 0, 0, 1, 5, 3, 8, ...>>`) to Kafka If `KAFKASERIALIZER` is not set to `avro`, producer sets **only** `cecontenttype` or `ce_contentType` for kafka event Event parsing is based on the implemented via . Check the section. In this example we'll have RIG send a message to itself to see whether RIG producing and consuming parts work correctly. The idea is that RIG produces a serialized event as a result to an HTTP request and, a few moments later, consumes that same event (and deserializes it correctly). ```bash KAFKAPORTPLAIN=17092 KAFKAPORTSSL=17093 HOST=localhost docker-compose -f integrationtests/kafkatests/docker-compose.yml up -d docker run --name rig \ -e KAFKA_BROKERS=kafka:9292 \ -e KAFKA_SERIALIZER=avro \ -e KAFKASCHEMAREGISTRY_HOST=kafka-schema-registry:8081 \ -e KAFKASOURCETOPICS=rigRequest \ -e PROXYCONFIGFILE='[{"id":"my-api","name":"my-api","versioned":false,"versiondata":{"default":{"endpoints":[{"id":"post-myapi-publish-async","pathregex":"/myapi/publish-async","method":"POST","target":"kafka","topic":"rigRequest","schema":"rigRequest-value"}]}},"proxy":{"useenv":true,"targeturl":"KAFKA_HOST","port":9092}}]' \ -e LOG_LEVEL=debug \ -p 4000:4000 -p 4010:4010 \ --network kafkatestsdefault \ accenture/reactive-interaction-gateway curl -d '{"schema":"{\"name\":\"rigproducer\",\"type\":\"record\",\"fields\":[{\"name\":\"example\",\"type\":\"string\"}]}"}' -H "Content-Type: application/vnd.schemaregistry.v1+json" -X POST http://localhost:8081/subjects/rigRequest-value/versions curl -d '{"id":"069711bf-3946-4661-984f-c667657b8d85","type":"com.example","time":"2018-04-05T17:31:00Z","specversion":"0.2","source":"\/cli","contenttype":"avro\/binary","data":{"example":"test"}}' -H "Content-Type: application/json" -X POST http://localhost:4000/myapi/publish-async 12:35:36.235 module=Avrora.Storage.Registry [debug] obtaining schema with global id `1` 12:35:36.236 module=RIG.Tracing [debug] private mode, remove tracestate. 12:35:36.237 module=Rig.EventStream.KafkaToFilter [debug] %{:struct => Cloudevents.Format.V02.Event, :contenttype => "avro/binary", :data => %{"example" => "test"}, :extensions => %{"rig" => %{"correlation" => "NTd2PR-TnLX00m4qYT-5fiSvuIymOvikja50kMPEYkezFkMcwhBY33rCgYCgKnlSSozVbYQZhWxqR8=", "headers" => [["accept", "*/*"], ["content-length", "188"], ["content-type", "application/json"], ["host", "localhost:4000"], ["user-agent", "curl/7.64.1"]], "host" => "localhost", "method" => "POST", "pathregex" => "/myapi/publish-async", "port" => 4000, "query" => "", "remoteip" => "172.19.0.1", "scheme" => "http"}, "traceparent" => "00-b068f0d83c5938fa6d0fb8c21959edf6-0ee83bcb44fb61e0-01"}, :id => "069711bf-3946-4661-984f-c667657b8d85", :schemaurl => nil, :source => "/cli", :specversion => "0.2", :time => "2018-04-05T17:31:00Z", :type => "com.example", "traceparent" => "00-86567e8c4b4cdb6c7f052b0f627959f9-c00644b7954af2fd-01"} ``` To check if it works also with native serializer we can leverage the CLI shipped with the Kafka Schema Registry image. ``` bash docker exec -it kafka-schema-registry bash kafka-avro-console-consumer --topic rigRequest \ --bootstrap-server kafka:9292 \ --property schema.registry.url='http://kafka-schema-registry:8081' curl -d '{"id":"069711bf-3946-4661-984f-c667657b8d85","type":"com.example","time":"2018-04-05T17:31:00Z","specversion":"0.2","source":"\/cli","contenttype":"avro\/binary","data":{"example":"test"}}' -H "Content-Type: application/json" -X POST http://localhost:4000/myapi/publish-async {"example":"test"} ```