bpot/poseidon — pull request #18 by bpot (2013-09-25 23:47:14+00:00)
Title: Producing a UTF-8 Message will cause the consumer to miss part of the message

Sorry for the long examples...

In Ruby 1.9.3 if I send the following EDN message

{:action :create, :object {"id" 1000000001, "name" "Jazztel", "website" "", "is_system_owner" nil, "password_expiry_interval" nil, "created_at" #inst "2013-02-01T19:15:37Z", "updated_at" #inst "2013-04-03T03:27:26Z", "logo_url" "htugny1murmnho3q9l4cogxr3fo61qtd", "address" "", "cenx_cma" "1792-1", "cenx_id" "100134666507", "notes" "", "terms" {:path "Path", :enni "ENNI", :network_type "Network Type", :aggregation_site "Aggregation Site", :site "Site", :demarc "Demarc", :site_group "Site Group", :service_provider "Service Provider", :segment "Segment", :wdc "WDC", :microwatt "µW", :aav "AAV", :gateway_site "CDC", :cascade "Cascade ID"}, "is_system_logo" false, "public_id" "ServiceProvider:Jazztel", "base_class" "ServiceProvider"}, :changed {}}

In this message there is one UTF-8 char :microwatt "µW" the µ is two bytes.
Using
$KAFKA/kafka-console-consumer.sh --zookeeper localhost --topic cenx.elm.inventory.updates --from-beginning

I see
{:action :create, :object {"id" 1000000001, "name" "Jazztel", "website" "", "is_system_owner" nil, "password_expiry_interval" nil, "created_at" #inst "2013-02-01T19:15:37Z", "updated_at" #inst "2013-04-03T03:27:26Z", "logo_url" "htugny1murmnho3q9l4cogxr3fo61qtd", "address" "", "cenx_cma" "1792-1", "cenx_id" "100134666507", "notes" "", "terms" {:path "Path", :enni "ENNI", :network_type "Network Type", :aggregation_site "Aggregation Site", :site "Site", :demarc "Demarc", :site_group "Site Group", :service_provider "Service Provider", :segment "Segment", :wdc "WDC", :microwatt "uW", :aav "AAV", :gateway_site "CDC", :cascade "Cascade ID"}, "is_system_logo" false, "public_id" "ServiceProvider:Jazztel", "base_class" "ServiceProvider"}, :changed {}

the LAST } is missing.

Looking at the code it seems the .size is being called to set the length in the message however if it's a UTF-8 the size will be the number of characters not the byte length (bytesize will return the correct byte size)

I have worked around this by forcing the EDN string to be ASCII-8BIT so the .size will return the correct size

Poseidon::MessageToSend.new(@@topic, data.to_edn.force_encoding("ASCII-8BIT"))