Package: cprotobuf
A minimal fast protobuf implementation with cython.
Benchmark shows that it’s much faster than google official expremental cpp-python implementation.I’ve been using it in production since 2013, only tested with python2.7, feedback on other python release is welcome.Benchmark$./setup.pybuild_ext--inplace$cdbenchmark$./bench.shencode[googleofficialpurepython]:10loops,bestof3:68.8msecperloopencode[googleofficialcpppython]:100loops,bestof3:19.4msecperloopencode[py-protobuf][cprotobuf]:100loops,bestof3:3.58msecperloopdecode[googleofficialpurepython]:10loops,bestof3:47.5msecperloopdecode[googleofficialcpppython]:100loops,bestof3:4.55msecperloopdecode[py-protobuf][cprotobuf]:100loops,bestof3:3.98msecperloopTutorialUse pluginYou write aperson.protofile like this:packagefoo;messagePerson{requiredint32id=1;requiredstringname=2;optionalstringemail=3;}And apeople.protofile like this:packagefoo;import"person.proto";messagePeople{repeatedPersonpeople=1;}Then you compile it with provided plugin:$protoc--cprotobuf_out=.person.protopeople.protoIf you have trouble to run a protobuf plugin like on windows, you can directly runprotoc-gen-cprotobuflike this:$protoc-ofoo.pbperson.protopeople.proto$protoc-gen-cprotobuffoo.pb-d.Then you get a python modulefoo_pb.py, cprotobuf generate a python module for each package rather than each protocol file.The generated code is quite readable:# coding: utf-8fromcprotobufimportProtoEntity,Field# file: person.protoclassPerson(ProtoEntity):id=Field('int32',1)name=Field('string',2)email=Field('string',3,required=False)# file: people.protoclassPeople(ProtoEntity):people=Field(Person,1,repeated=True)Actually, if you only use python, you can write this python module, avoid code generation.The APINow, you have this lovely python module, how to parse and serialize messages?When design this package, We try to minimise the effort of migration, so we keep the names of api akin to protocol buffer’s.NoteSince this is no need to reuse a message instance and callClearon it in python, It don’t provideClearapi,
soParseFromStringis more likeMergeFromStringin official implementation, because it don’t callClearat first.encode/decode>>>fromfoo_pbimportPerson,People>>>msg=People()>>>msg.people.add(...id=1,...name='jim',...email='jim@gmail.com',...)>>>s=msg.SerializeToString()>>>msg2=People()>>>msg2.ParseFromString(s)>>>len(msg2)1>>>msg2.people[0].name'jim'reflection>>>fromfoo_pbimportPerson,People>>>dir(Person._fields[0])['__class__','__delattr__','__doc__','__format__','__get__','__getattribute__','__hash__','__init__','__new__','__pyx_vtable__','__reduce__','__reduce_ex__','__repr__','__setattr__','__sizeof__','__str__','__subclasshook__','index','name','packed','repeated','required','wire_type']>>>Person._fields[0].name'email'>>>Person._fieldsmap{1:<cprotobuf.Fieldobjectat0xb74a538c>,2:<cprotobuf.Fieldobjectat0xb74a541c>,3:<cprotobuf.Fieldobjectat0xb74a5c8c>}>>>Person._fieldsmap_by_name{'email':<cprotobuf.Fieldobjectat0xb74a5c8c>,'name':<cprotobuf.Fieldobjectat0xb74a541c>,'id':<cprotobuf.Fieldobjectat0xb74a538c>}repeated containerWe useRepeatedContainerto represent repeated field,RepeatedContaineris inherited fromlist, so you can manipulate it like alist, or with apis like google’s implementation.>>>fromfoo_pbimportPerson,People>>>msg=People()>>>msg.people.add(...id=1,...name='jim',...email='jim@gmail.com',...)>>>p=msg.people.add()>>>p.id=2>>>p.name='jake'>>>p.email='jake@gmail.com'>>>p2=Person(id=3,name='lucy',email='lucy@gmail.com')>>>msg.people.append(p2)>>>msg.people.append({...'id':4,...'name':'lily',...'email':'lily@gmail.com',...})encode raw data fastIf you already have your messages represented aslistanddict, you can encode it without constructing intermidiate objects, getting ride of a lot of overhead:>>>fromcprotobufimportencode_data>>>fromfoo_pbimportPerson,People>>>s=encode_data(People,[...{'id':1,'name':'tom','email':'tom@gmail.com'}...])>>>msg=People()>>>msg.ParseFromString(s)>>>msg.people[0].name'tom'Utility APIs>>>fromcprotobufimportencode_primitive,decode_primitive>>>encode_primitive('uint64',10)bytearray(b'\x01')>>>decode_primitive(b'\n','uint64')(10,1)Run Tests$ nosetests