Package: python-proto-converter
Python Proto ConverterThe Python Proto Converter converts between protos in Python. Proto conversion is often needed when converting between Database Access Object (DAO) and API proto.Installpip install python-proto-converterRun the exampleBuild the proto (assuming in exmaple/ directory)
protoc -I=. --python_out=. ./example_proto.protoexecute
python3 ./converter_example.pyFeaturesA base class that auto-converts fields with the same name and type.Custom convert functions can be implemented to handle fields conversion.Fields can be disabled during auto-converting.Unhandled fields assertion during class instantiation.ExampleBasic usageLet's start with a simple example, suppose you want to convert from one similar
proto to another. For this example, these are the MatchaMilkTea to
GreenTeaMilkTea protos.messageMatchaMilkTea{stringname=1;floatprice=2;stringseller=3;}messageGreenTeaMilkTea{stringname=1;int64price=2;stringseller=3;}Thenameandsellerfields can be auto-converted, since the type and the
field name are identical. However, we probably don't want to copy the name of
MatchaMilkTea to GreenTeaMilkTea. To disable auto-convert on thenamefield,
we mark it ignored and provide our custom function for thenamefield.Thepricefield has different types (float vs int64), therefore it can't be
auto-converted. Leaving it unhandled will trigger an exception when creating the
proto converter. Similar to thenamefield, we can create a custom method to
convert thepricefield.fromgoogle3.alkali.contrib.certified.python.protoimportconverter...classMatchaToGreenTeaConverter(converter.ProtoConverter):def__init__(self):super(MatchaToGreenTeaConverter,self).__init__(pb_class_from=matcha_milk_tea_pb2.MatchaMilkTea,pb_class_to=green_tea_milk_tea_pb2.GreenTeaMilkTea,field_names_to_ignore=["name"])@converter.convert_field(field_names=["price"])defprice_convert_function(self,src_proto,dest_proto):dest_proto.price=int(src_proto.price)@converter.convert_field(field_names=["name"])defname_convert_function(self,src_proto,dest_proto):dest_proto.name="GreenTeaMilkTea"Or you can combine them in the same method since these fields are simple:@converter.convert_field(field_names=["price","name"])defprice_name_convert_function(self,src_proto,dest_proto):dest_proto.price=int(src_proto.price)dest_proto.name="GreenTeaMilkTea"Now you can create the converter in code and use it:...matcha_to_green_tea_converter=MatchaToGreenTeaConverter()green_tea_milk_tea_proto=matcha_to_green_tea_converter.convert(matcha_milk_tea_proto)...Nested protosLet's make this example a bit more complicated by adding some fields.enumFlavor{GREEN_TEA=0;MATCHA=1;BERRY=2;SPICY=3;}messageMilkTea{stringname=1;floatprice=2;Flavorflavor=3;}messageMatchaMilkTea{MilkTeamilk_tea=1;int64sugar=2;repeatedstringshops=3;stringmatcha_provider=4;map<string,int64>ingredients=5;map<string,string>ingredients_calorie_map=6;repeatedstringcup_sizes=7;}messageGreenTeaMilkTea{MilkTeamilk_tea=1;floatsugar=2;repeatedstringshops=3;stringgreen_tea_provider=4;map<string,int64>ingredients=5;map<string,int32>ingredients_calorie_map=6;repeatedint64cup_sizes=7;}Most of the fields are identical and can be auto-converted, except:float sugar and int64 sugar;string green_tea_provider;string matcha_provider;ingredients_calorie_map;cup_sizes;You can create a new MatchaToGreenTeaConverter class that inherits ProtoConverter
to convert from MatchaMilkTea to GreenTeaMilkTea:fromgoogle3.alkali.contrib.certified.python.protoimportconverter...classMatchaToGreenTeaConverter(converter.ProtoConverter):def__init__(self):super(MatchaToGreenTeaConverter,self).__init__(pb_class_from=matcha_milk_tea_pb2.MatchaMilkTea,pb_class_to=green_tea_milk_tea_pb2.GreenTeaMilkTea,field_names_to_ignore=["ingredients_calorie_map","cup_sizes"])@converter.convert_field(field_names=["sugar"])defsugar_convert_function(self,src_proto,dest_proto):dest_proto.sugar=int(src_proto.sugar)@converter.convert_field(field_names=["matcha_provider"])defprovider_convert_function(self,src_proto,dest_proto):dest_proto.green_tea_provider=src_proto.matcha_providerpb_class_fromandpb_class_toare the constructors of the protos.pb_class_from.Fields infield_names_to_ignorewill be ignored during
auto-conversion and when validating that all fields have been handled. In
the example,ingredients_calorie_mapandcup_sizesare ignored during
conversion.@converter.convert_fielddecorates a custom conversion function. In this
example, we have two functions to convert thesugarfield and thematcha_providerfield.All fields that can't be auto-converted from the source proto must either be
handled by custom conversion functions or listed infield_names_to_ignore.Oneof fieldsOneof fields can be tricky and error-prone, therefore it is required to
explicitly handle or ignore all the fields in oneofs.messageMochiFlavor{stringflavor=1;}messageMochi{oneofprice{stringprice_str=1;floatprice_float=2;}oneofflavor{Flavorflavor_enum=3;MochiFlavorflavor_proto=4;}int64calorie=5;}messageTaroMochi{floatprice_float=1;MochiFlavorflavor_proto=2;int64calorie=3;}proto_converter=converter.ProtoConverter(pb_class_from=mochi_pb2.Mochi,pb_class_to=mochi_pb2.Taromochi,field_names_to_ignore=["flavor_enum","price_str"])src_proto=mochi_pb2.Mochi(price_float=3.14,flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),calorie=100)dest_proto=proto_converter.convert(src_proto=src_proto)In the above example, even thoughflavor_enumandprice_strfields are not
used, ProtoConverter will still raise an exception if these fields are not
ignored.Any fieldsPrototoAnyandAnytoAnyare converted automatically as long as the
field name matches.messageAnyMochiBox{stringname=1;google.protobuf.Anymochi=2;}messageTaroMochiBox{stringname=1;TaroMochimochi=2;}In the example below, ProtoConverter auto-converts a TaroMochi field to a Any
field.taro_mochi=mochi_pb2.TaroMochi(price_float=3.14,flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),calorie=100)proto_converter=converter.ProtoConverter(pb_class_from=mochi_pb2.TaroMochiBox,pb_class_to=mochi_pb2.AnyMochiBox)src_proto=mochi_pb2.TaroMochiBox(name="TaroMochiBox",mochi=taro_mochi)dest_proto=proto_converter.convert(src_proto=src_proto)Similarily, ProtoConverter auto-converts Proto Any field to Any field.taro_mochi=mochi_pb2.TaroMochi(price_float=3.14,flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),calorie=100)taromochi_any_proto=any_pb2.Any()taromochi_any_proto.Pack(taro_mochi)proto_converter=converter.ProtoConverter(pb_class_from=mochi_pb2.AnyMochiBox,pb_class_to=mochi_pb2.AnyMochiBox)src_proto=mochi_pb2.AnyMochiBox(name="TaroMochiBox",mochi=taromochi_any_proto)dest_proto=proto_converter.convert(src_proto=src_proto)RepeatedAnyfield and MapAnyfield are also supported.messageAnyMochiBoxes{stringname=1;repeatedgoogle.protobuf.Anymochi=2;}messageTaroMochiBoxes{stringname=1;repeatedTaroMochimochi=2;}messageMochiGiftPackage{stringname=1;map<string,google.protobuf.Any>mochi=2;}messageTaroMochiGiftPackage{stringname=1;map<string,google.protobuf.Any>mochi=2;}The examples below demonstrate the auto-conversion for repeated fields and Map
fields with Any proto.proto_converter=converter.ProtoConverter(pb_class_from=mochi_pb2.TaroMochiBoxes,pb_class_to=mochi_pb2.AnyMochiBoxes)src_proto=mochi_pb2.TaroMochiBoxes(name="TaroMochiBoxes",mochi=[taro_mochi,taro_mochi])dest_proto=proto_converter.convert(src_proto=src_proto)proto_converter=converter.ProtoConverter(pb_class_from=mochi_pb2.TaroMochiGiftPackage,pb_class_to=mochi_pb2.AnyMochiGiftPackage)src_proto=mochi_pb2.TaroMochiGiftPackage(name="TaroMochiBoxes",mochi={"taro_mochi":taro_mochi})dest_proto=proto_converter.convert(src_proto=src_proto)We decided not to supportAnyfield toProtofield auto conversion to make
it less error-pone, since theAnyfield can contain any type and cause runtime
failures. However, it is very easy to add a custom method to handleAnyfield.classMochiConverter(converter.ProtoConverter):@converter.convert_field(field_names=["mochi"])defmochi_field_convert_function(self,src_proto,dest_proto):src_proto.mochi.Unpack(dest_proto.mochi)...taro_mochi=mochi_pb2.TaroMochi(price_float=3.14,flavor_proto=mochi_pb2.MochiFlavor(flavor="taro"),calorie=100)taromochi_any_proto=any_pb2.Any()taromochi_any_proto.Pack(taro_mochi)proto_converter=MochiConverter(pb_class_from=mochi_pb2.AnyMochiBox,pb_class_to=mochi_pb2.TaroMochiBox)src_proto=mochi_pb2.AnyMochiBox(name="TaroMochiBox",mochi=_pack_to_any_proto(taro_mochi))dest_proto=proto_converter.convert(src_proto=src_proto)RepeatedAnyfield to repeatedProtofieldclassRepeatedMochiConverter(converter.ProtoConverter):@converter.convert_field(field_names=["mochi"])defmochi_field_convert_function(self,src_proto,dest_proto):forfieldinsrc_proto.mochi:proto_object=mochi_pb2.TaroMochi()field.Unpack(proto_object)dest_proto.mochi.append(proto_object)MapAnyfield to MapProtofieldclassMapMochiConverter(converter.ProtoConverter):@converter.convert_field(field_names=["mochi"])defmochi_field_convert_function(self,src_proto,dest_proto):forkey,fieldinsrc_proto.mochi.items():proto_object=mochi_pb2.TaroMochi()field.Unpack(proto_object)dest_proto.mochi[key].CopyFrom(proto_object)Nested conversionNested conversion is supported if the source proto and destination proto
contains the same proto type (like the above example),
while auto-conversion won't work if the nested protos are of different type.However, it's very easy to support this case with a custom method. We think it's
cleaner to create separate converters as you will see in the below example.messageTaroMochi{floatprice_float=1;MochiFlavorflavor_proto=2;int64calorie=3;}messageCocoMochi{floatprice_float=1;MochiFlavorflavor_proto=2;int64calorie=3;}messageTaroMochiBox{stringname=1;TaroMochimochi=2;}messageCocoMochiBox{stringname=1;CocoMochimochi=2;}classNestedMochiBoxConverter(converter.ProtoConverter):taro_to_coco_converter:converter.ProtoConverter=Nonedef__init__(self):super(RecursiveMochiBoxConverter,self).__init__(pb_class_from=mochi_pb2.TaroMochiBox,pb_class_to=mochi_pb2.CocoMochiBox)self.taro_to_coco_converter=converter.ProtoConverter(pb_class_from=mochi_pb2.TaroMochi,pb_class_to=mochi_pb2.CocoMochi)@converter.convert_field(field_names=["mochi"])defmochi_field_convert_function(self,src_proto,dest_proto):dest_proto.mochi.CopyFrom(self.taro_to_coco_converter.convert(src_proto.mochi))...proto_converter=NestedMochiBoxConverter()dest_proto=proto_converter.convert(src_proto)With the additional ProtoConverter between TaroMochi and CocoMochi, it's very
easy to update the conversion once the TaroMochi or CocoMochi proto changes.For nested array protos, we need to iterate through each element and append the
conversion result to the destination proto:messageCocoMochiBoxes{stringname=1;repeatedCocoMochimochi=2;}messageTaroMochiBoxes{stringname=1;repeatedTaroMochimochi=2;}@converter.convert_field(field_names=["mochi"])defmochi_field_convert_function(self,src_proto,dest_proto):formochiinsrc_proto.mochi:dest_proto.mochi.append(self.taro_to_coco_converter.convert(mochi))ContributingSeeCONTRIBUTING.mdfor details.LicenseApache 2.0; seeLICENSEfor details.DisclaimerThis project is not an official Google project. It is not supported by
Google and Google specifically disclaims all warranties as to its quality,
merchantability, or fitness for a particular purpose.