Package: betterproto
Better Protobuf / gRPC Support for PythonThis project aims to provide an improved experience when using Protobuf / gRPC in a modern Python environment by making use of modern language features and generating readable, understandable, idiomatic Python code. It will not support legacy features or environments (e.g. Protobuf 2). The following are supported:Protobuf 3 & gRPC code generationBoth binary & JSON serialization is built-inPython 3.6+ making use of:EnumsDataclassesasync/awaitTimezone-awaredatetimeandtimedeltaobjectsRelative importsMypy type checkingThis project is heavily inspired by, and borrows functionality from:https://github.com/protocolbuffers/protobuf/tree/master/pythonhttps://github.com/eigenein/protobuf/https://github.com/vmagamedov/grpclibMotivationThis project exists because I am unhappy with the state of the official Google protoc plugin for Python.Noasyncsupport (requires additionalgrpclibplugin)No typing support or code completion/intelligence (requires additionalmypyplugin)No__init__.pymodule files get generatedOutput is not importableImport paths break in Python 3 unless you mess withsys.pathBugs when names clash (e.g.codecspackage)Generated code is not idiomaticCompletely unreadable runtime code-generationMuch code looks like C++ or Java ported 1:1 to PythonCapitalized function names likeHasField()andSerializeToString()UsesSerializeToString()rather than the built-in__bytes__()Special wrapped types don't use Python'sNoneTimestamp/duration types don't use Python's built-indatetimemoduleThis project is a reimplementation from the ground up focused on idiomatic modern Python to help fix some of the above. While it may not be a 1:1 drop-in replacement due to changed method names and call patterns, the wire format is identical.Installation & Getting StartedFirst, install the package. Note that the[compiler]feature flag tells it to install extra dependencies only needed by theprotocplugin:# Install both the library and compiler$pipinstall"betterproto[compiler]"# Install just the library (to use the generated code output)$pipinstallbetterprotoNow, given you installed the compiler and have a proto file, e.gexample.proto:syntax="proto3";packagehello;// Greeting represents a message you can tell a user.messageGreeting{stringmessage=1;}You can run the following:$protoc-I.--python_betterproto_out=.example.protoThis will generatehello.pywhich looks like:# Generated by the protocol buffer compiler.  DO NOT EDIT!# sources: hello.proto# plugin: python-betterprotofromdataclassesimportdataclassimportbetterproto@dataclassclassHello(betterproto.Message):"""Greeting represents a message you can tell a user."""message:str=betterproto.string_field(1)Now you can use it!>>>fromhelloimportHello>>>test=Hello()>>>testHello(message='')>>>test.message="Hey!">>>testHello(message="Hey!")>>>serialized=bytes(test)>>>serializedb'\n\x04Hey!'>>>another=Hello().parse(serialized)>>>anotherHello(message="Hey!")>>>another.to_dict(){"message":"Hey!"}>>>another.to_json(indent=2)'{\n"message": "Hey!"\n}'Async gRPC SupportThe generated ProtobufMessageclasses are compatible withgrpclibso you are free to use it if you like. That said, this project also includes support for async gRPC stub generation with better static type checking and code completion support. It is enabled by default.Given an example like:syntax="proto3";packageecho;messageEchoRequest{stringvalue=1;// Number of extra times to echouint32extra_times=2;}messageEchoResponse{repeatedstringvalues=1;}messageEchoStreamResponse{stringvalue=1;}serviceEcho{rpcEcho(EchoRequest)returns(EchoResponse);rpcEchoStream(EchoRequest)returns(streamEchoStreamResponse);}You can use it like so (enable async in the interactive shell first):>>>importecho>>>fromgrpclib.clientimportChannel>>>channel=Channel(host="127.0.0.1",port=1234)>>>service=echo.EchoStub(channel)>>>awaitservice.echo(value="hello",extra_times=1)EchoResponse(values=["hello","hello"])>>>asyncforresponseinservice.echo_stream(value="hello",extra_times=1)print(response)EchoStreamResponse(value="hello")EchoStreamResponse(value="hello")JSONBoth serializing and parsing are supported to/from JSON and Python dictionaries using the following methods:Dicts:Message().to_dict(),Message().from_dict(...)JSON:Message().to_json(),Message().from_json(...)For compatibility the default is to convert field names tocamelCase. You can control this behavior by passing a casing value, e.g:>>>MyMessage().to_dict(casing=betterproto.Casing.SNAKE)Determining if a message was sentSometimes it is useful to be able to determine whether a message has been sent on the wire. This is how the Google wrapper types work to let you know whether a value is unset, set as the default (zero value), or set as something else, for example.Usebetterproto.serialized_on_wire(message)to determine if it was sent. This is a little bit different from the official Google generated Python code, and it lives outside the generatedMessageclass to prevent name clashes. Note that itonlysupports Proto 3 and thus canonlybe used to check ifMessagefields are set. You cannot check if a scalar was sent on the wire.# Old way (official Google Protobuf package)>>>mymessage.HasField('myfield')# New way (this project)>>>betterproto.serialized_on_wire(mymessage.myfield)One-of SupportProtobuf supports grouping fields in aoneofclause. Only one of the fields in the group may be set at a given time. For example, given the proto:syntax="proto3";messageTest{oneoffoo{boolon=1;int32count=2;stringname=3;}}You can usebetterproto.which_one_of(message, group_name)to determine which of the fields was set. It returns a tuple of the field name and value, or a blank string andNoneif unset.>>>test=Test()>>>betterproto.which_one_of(test,"foo")["",None]>>>test.on=True>>>betterproto.which_one_of(test,"foo")["on",True]# Setting one member of the group resets the others.>>>test.count=57>>>betterproto.which_one_of(test,"foo")["count",57]>>>test.onFalse# Default (zero) values also work.>>>test.name="">>>betterproto.which_one_of(test,"foo")["name",""]>>>test.count0>>>test.onFalseAgain this is a little different than the official Google code generator:# Old way (official Google protobuf package)>>>message.WhichOneof("group")"foo"# New way (this project)>>>betterproto.which_one_of(message,"group")["foo","foo's value"]Well-Known Google TypesGoogle provides several well-known message types like a timestamp, duration, and several wrappers used to provide optional zero value support. Each of these has a special JSON representation and is handled a little differently from normal messages. The Python mapping for these is as follows:Google MessagePython TypeDefaultgoogle.protobuf.durationdatetime.timedelta0google.protobuf.timestampTimezone-awaredatetime.datetime1970-01-01T00:00:00Zgoogle.protobuf.*ValueOptional[...]NoneFor the wrapper types, the Python type corresponds to the wrapped type, e.g.google.protobuf.BoolValuebecomesOptional[bool]whilegoogle.protobuf.Int32ValuebecomesOptional[int]. All of the optional values default toNone, so don't forget to check for that possible state. Given:syntax="proto3";import"google/protobuf/duration.proto";import"google/protobuf/timestamp.proto";import"google/protobuf/wrappers.proto";messageTest{google.protobuf.BoolValuemaybe=1;google.protobuf.Timestampts=2;google.protobuf.Durationduration=3;}You can do stuff like:>>>t=Test().from_dict({"maybe":True,"ts":"2019-01-01T12:00:00Z","duration":"1.200s"})>>>tTest(maybe=True,ts=datetime.datetime(2019,1,1,12,0,tzinfo=datetime.timezone.utc),duration=datetime.timedelta(seconds=1,microseconds=200000))>>>t.ts-t.durationdatetime.datetime(2019,1,1,11,59,58,800000,tzinfo=datetime.timezone.utc)>>>t.ts.isoformat()'2019-01-01T12:00:00+00:00'>>>t.maybe=None>>>t.to_dict(){'ts':'2019-01-01T12:00:00Z','duration':'1.200s'}DevelopmentFirst, make sure you have Python 3.6+ andpipenvinstalled, along with the officialProtobuf Compilerfor your platform. Then:# Get set up with the virtual env & dependencies$pipenvinstall--dev# Link the local package$pipenvshell
$pipinstall-e.Code styleThis project enforcesblackpython code formatting.Before commiting changes run:pipenvrunblack.To avoid merge conflicts later, non-black formatted python code will fail in CI.TestsThere are two types of tests:Standard testsCustom testsStandard testsAdding a standard test case is easy.Create a new directorybetterproto/tests/inputs/<name>add<name>.protowith a message calledTestadd<name>.jsonwith some test dataIt will be picked up automatically when you run the tests.See also:Standard Tests Development GuideCustom testsCustom tests are found intests/test_*.pyand are run with pytest.RunningHere's how to run the tests.# Generate assets from sample .proto files$pipenvrungenerate# Run all tests$pipenvruntestTODOFixed length fieldsPacked fixed-lengthZig-zag signed fields (sint32, sint64)Don't encode zero values for nested typesEnumsRepeated message fieldsMapsMaps of message fieldsSupport passthrough of unknown fieldsRefs to nested typesImports in proto filesWell-known Google typesSupport as request inputSupport as response outputAutomatically wrap/unwrap responsesOneOf supportBasic support on the wireCheck which was set from the groupSetting one unsets the othersJSON that isn't completely naive.64-bit ints as stringsMapsListsBytes as base64Any supportEnum stringsWell known types support (timestamp, duration, wrappers)Support different casing (orig vs. camel vs. others?)Async service stubsUnary-unaryServer streaming responseClient streaming requestRenaming messages and fields to conform to Python name standardsRenaming clashes with language keywordsPython packageAutomate running testsCleanup!LicenseCopyright © 2019 Daniel G. Taylorhttp://dgt.mit-license.org/