Package: django-pb-model
Django-pb-model provides model mixin mapping/converting protobuf message.
Automatic model generation from protobuf message definitions is supported.
Currently support basic value fields and naive relation conversion, including:Integer, String, Float, BooleanChoices fieldDatetimeForeign Key and Many-to-Many relationCustom fields, ex: JSONYou could examinetestcasesfor more detailsAnd PRs are always welcome :))Table of ContentCompatibilityInstallUsageAutomatic field generationField detailsField name mappingForeign KeyMany-to-Many fieldDjango to ProtobufProtobuf to DjangoLimit Foreign key or Many-to-Many field conversion depthDatetime FieldTimezoneAnyCustom FieldsBuilt-InsCompatibilityCurrently tested with matrix:Django/Python2.73.53.63.73.81.11.xv2.2.xvvv3.0.xvvvInstallpip installpipinstalldjango-pb-modelAdd django-pb to djangosettings.pyINSTALLED_APPS=[....,pb_model,...]Run python/django essential commands:pythonmanage.pymakemigrationspythonmanage.pymigratepythonmanage.pycollectstatic--linkStart hacking or using app.UsageDeclare your protobuf message file, such asaccount.proto, and compile it. For example:messageAccount{intid=1;stringemail=2;stringnickname=3;}Then compile it with:$protoc--python_out=.account.protoYou will getaccount_pb2.py.Now you can interact with your protobuf model, addProtoBufMixinto your model like:fromdjango.dbimportmodelsfrompb_model.modelsimportProtoBufMixinfrom.importaccount_pb2classAccount(ProtoBufMixin,models.Model):pb_model=account_pb2.Accountemail=models.EmailField(max_length=64)nickname=models.CharField(max_length=64)def__str__(self):return"Username:{a.email}, nickname:{a.nickname}".format(a=self)By above settings, you can convert between django model and protobuf easily. For example:>>>account=Account.objects.create(email='user@email.com',nickname='moonmoon')>>>account.to_pb()email:"user@email.com"nickname:"moonmoon">>>account2=Account()>>>account2.from_pb(account.to_pb())<Account:Username:username@mail,nickname:moonmoon>Automatic field generationTo automatically generate django model fields based on protobuf field types.If you don’t want to manually specify fields in your django model, you can list names of desired fields underpb_2_dj_fieldsattribute to have those generated and added to your model automatically.classAccount(ProtoBufMixin,models.Model):pb_model=account_pb2.Accountpb_2_dj_fields=['email','nickname']Alternatively if you want all protobuf fields to be mapped you can dopb_2_dj_fields = '__all__'.Fields listed inpb_2_dj_fieldscan be overwritten using manual definition.classAccount(ProtoBufMixin,models.Model):pb_model=account_pb2.Accountpb_2_dj_fields='__all__'email=models.EmailField(max_length=64)Type of generated field depends on corresponding protobuf field type. If you want to change default field type mappings you can overwrite those usingpb_auto_field_type_mappingattribute.Following protobuf field types are supported:uint32, int32, uint64, int64, float, double, bool, Enumstring, bytesgoogle.protobuf.Timestampgoogle.protobuf.AnyMessagesoneof fieldsrepeated scalar and Message fieldsmap fields with scalar as key and scalar or Message as valueField detailsThere are several special field types while converting, read following sections.Field name mappingTo adapt schema migration, field mapping are expected.For example, theemailfield in previous session is altered tousername, but we don’t want to break the consistence of protobuf protocol. You may addpb_2_dj_field_mapattribute to solve this problem. Such as:classAccount(ProtoBufMixin,models.Model):pb_model=account_pb2.Accountpb_2_dj_field_map={"email":"username",# protobuf field as key and django field as value}username=models.CharField(max_length=64)nickname=models.CharField(max_length=64)Foreign KeyForeign key is a connect to another model in Django. According to this property, the foreign key could and should be converted to nested singular message in Protobuf. For example:messageRelation{int32id=1;}messageMain{int32id=1;Relationfk=2;}Django model:classRelation(ProtoBufMixin,models.Model):pb_model=models_pb2.RelationclassMain(ProtoBufMixin,models.Model):pb_model=models_pb2.Mainfk=models.ForeignKey(Relation)With above settings, pb_model would recursively serialize and de-serialize between Django and ProtoBuf.>>>m=Main.objects.create(fk=Relation.objects.create())>>>m.to_pb()id:1fk{id:1}>>>m2=Main()>>>m2.from_pb(m.to_pb())>>>m2.fk.id1Note that one can specify a reversed relation by assign related_name:classRelation(ProtoBufMixin,models.Model):pb_model=models_pb2.Relationnum=models.IntegerField(default=0)deeper_relation=models.ForeignKey(DeeperRelation,on_delete=models.DO_NOTHING,blank=True,null=True,related_name='relations')When the related proto contains the same field of this reversed relation:messageDeeperRelation{int32id=1;int32num=2;repeatedRelationrelations=3;}we will skip serializes the relations field.Many-to-Many fieldM2M field is a QuerySet Relation in Django.
By default, we assume target message field is “repeated” nested message, ex:messageM2M{int32id=1;}messageMain{int32id=1;repeatedM2Mm2m=2;}Django model would be:classM2M(models.Model):passclassMain(models.Model):m2m=models.ManyToManyField(M2M)Django to ProtobufIf this is not the format you expected, overwrite_m2m_to_protobuf()of Django model by yourself.Protobuf to DjangoSame as previous section, we assume m2m field is repeated value in protobuf.
By default,NOoperation is performed, which means
you may query current relation if your converted django model instance has a valid primary key.If you want to modify your database while converting on-the-fly, overwrite
logics such as:fromdjango.dbimporttransaction...classPBCompatibleModel(ProtoBufMixin,models.Model):def_repeated_to_m2m(self,dj_field,_pb_repeated_set):withtransaction.atomic():foritemin_pb_repeated_set:dj_field.get_or_create(pk=item.pk,defaults={....})...Also, you should write your converting policy if m2m is not nested repeated message in_repeated_to_m2mmethodLimit Foreign key or Many-to-Many field conversion depthBy default, when to_pb() method is called, all related message will be
also converted recursively.For example:messageDeeperRelation{int32id=1;int32num=2;}// Relation model for testingmessageRelation{int32id=1;int32num=2;DeeperRelationdeeper_relation=3;}messageMain{int32id=1;Relationfk_field=2;}And code:>>>m=Main.objects.create(fk=Relation.objects.create(deeper_relation=DeeperRelation.objects.create()))>>>m.to_pb()fk{id:1fk_field{id:1,deeper_relation{id:1}}}This may not be the behavior wanted. The depth parameter can be used to limit
the depth of these conversion.If the depth is set to 0, no related field will be converted, the fk_field in
Main message will left unset.If the depth is set to any positive number, the level of related field will be
limited by the specified number. For example, if depth is set to 1, the fk_field
will contain the related Relation message, however the deeper_relation field
of the fk_field message will be unset.Datetime FieldDatetime is a special singular value and able to convert betweendatetime.datetime(Python) andgoogle.protobuf.timestamp_pb2.Timestamp(ProboBuf)
through built-in datetime serializers. CheckCustom Fieldsif you want other coversion rules.Default conversion works as following example:ProtoBuf message:packagemodels;import"google/protobuf/timestamp.proto";messageWithDatetime{int32id=1;google.protobuf.Timestampdatetime_field=2;}Django Model:classWithDatetime(ProtoBufMixin,models.Model):pb_model=models_pb2.WithDatetimedatetime_field=models.DatetimeField(default=timezone.now())>>>WithDatetime.objects.create().to_pb()datetime_field{seconds:1495119614nanos:282705000}TimezoneNote that if you useUSE_TZin Django settings, all datetime would be converted to UTC timezone while storing in protobuf message.
And converted to default timezone in django according to settings.AnyWhen usingAnyas a field in a message, the field is by default kept as Any in your Django model and you may save it like it is any other Django Field.
Example on usingAny:packagemodels;import"google/protobuf/any.proto";messageWithAnyMessage{google.protobuf.Anyany_field=1;}classWithAny(ProtoBufMixin,models.Model):pb_model=models_pb2.WithAnyThen when you’re using your Django:>>>with_any_message=WithAnyMessage(any_field=Any())>>>WithAny().from_pb(with_any_message).save()# See that the we successfully saved Any as a Django field.>>>any_field=WithAny.objects.last().any_field>>>type(any_field)google.protobuf.any_pb2.AnyCustom FieldsYou can write your own field serializers, to convert betweendjango.contrib.postgres.fields.JSONField(Python)
andstring(Protobuf) for example:ProtoBuf message:packagemodels;messageWithJSONBlob{int32id=1;stringjson_blob=2;}Django Model:defjson_serializer(pb_obj,pb_field,dj_value):setattr(pb_obj,pb_field.name,json.dumps(value))defjson_deserializer(instance,dj_field_name,pb_field,pb_value):setattr(instance,dj_field_name,json.loads(pb_value))classWithJSONField(ProtoBufMixin,models.Model):pb_model=models_pb2.WithJSONBlobpb_2_dj_field_serializers={'json_blob':(json_serializer,json_deserializer),}json_blob=JSONField()Built-InsThere are 2 built-in serializers for types:django.models.UUIDFieldanddjango.models.DateTimeField._pb_2_dj_default_field_serializers={models.DateTimeField:(fields._datetimefield_to_pb,fields._datetimefield_from_pb),models.UUIDField:(fields._uuid_to_pb,fields._uuid_from_pb),}And is able to be override by declaration inpb_2_dj_field_serializers.