Package: protobuf-to-pydantic
protobuf_to_pydanticGenerate Pydantic Model or source code with parameter verification function based on Protobuf file (Proto3).中文文档FeatureFeature：Generate source code throughProtobufplugin。GeneratePydantic Modelor source code by parsingProtobuf MessageinPythonruntime.Compatible withV1andV2versions ofPydantic。Supports multiple verification rules and is compatible withproto-gen-validate(subsequent versions will support the rules ofproto-gen-validate1.0)。Support custom functionality through templates。Supportsprotovalidateverification rules（proto-gen-validateversion >= 1.0）The following is a functional overview diagram ofprotobuf-to-pydantic.
In the pictureP2Prepresentsprotobuf-to-pydantic,Protocrepresents the command forProtobufto generate code, andpluginrepresentsPlugin for Protoc:InstallationBy default,protobuf-to-pydanticcan be installed directly via the following command:pipinstallprotobuf_to_pydanticIf want to use the full functionality ofprotobuf-to-pydantic, can installprotobuf-to-pydanticwith the following command:.pipinstallprotobuf_to_pydantic[all]Usage1.code generationprotobuf-to-pydanticcurrently has two methods to generatePydantic Modelobjects based on Protobuf files.：1: Use theProtocplug-in to generate the correspondingPythoncode file through the Protobuf file。2: Generate the correspondingPydantic Modelobject through theMessageobject inPythonruntime。1.1.Directly generatePydantic Modelcode files through plug-ins1.1.0.Install dependenciesTheprotobuf-to-pydanticplug-in depends onmypy-protobuf, need to installmypy-protobufthrough the following command first:python-mpipinstallprotobuf-to-pydantic[mypy-protobuf]orpoetryaddprotobuf-to-pydantic-Emypy-protobuf1.1.1.Use pluginsPlug-in is thePydantic Modelsource code generation method recommended byprotobuf-to-pydantic.
It supports the most complete functions and is also very simple to use.Assume that it is usually generated through the following command Code corresponding to Protobuf file:python-mgrpc_tools.protoc-I.example.protoAfter installingprotobuf-to-pydantic,can use theprotobuf-to-pydanticplugin with the--protobuf-to-pydantic_outoption with the following command:python-mgrpc_tools.protoc-I.--protobuf-to-pydantic_out=.example.protoIn this command,--protobuf-to-pydantic_out=.means using theprorobuf-to-pydanticplug-in,
And it is declared that the output location of theprotobuf-to-pydanticplug-in is..indicates the output path used bygrpc_tools.proto.After running the command, theprotobuf-to-pydanticplugin writes the generated source code to a file with the filename suffixp2p.py, e.g.,example.protogenerates a file with the nameexample_p2p.py.1.1.2.Plug-in configurationTheprotobuf-to-pydanticplugin supports loading configuration by reading aPythonfile。In order to ensure that the variables of the configuration file can be introduced normally, the configuration file must be stored in the current path of the running command.。An example configuration that can be read byprotobuf-to-pydanticis as follows:importloggingfromtypingimportList,Typefromgoogle.protobuf.any_pb2importAny# type: ignorefrompydanticimportconfloat,conintfrompydantic.fieldsimportFieldInfofromprotobuf_to_pydantic.desc_templateimportDescTemplate# Configure the log output format and log level of the plugin, which is very useful when debugginglogging.basicConfig(format="[%(asctime)s%(levelname)s]%(message)s",datefmt="%y-%m-%d%H:%M:%S",level=logging.DEBUG)classCustomerField(FieldInfo):passdefcustomer_any()->Any:returnAny# type: ignore# For the configuration of the local template, see the use of the local template for detailslocal_dict={"CustomerField":CustomerField,"confloat":confloat,"conint":conint,"customer_any":customer_any,}# Specifies the start of key commentscomment_prefix="p2p"# Specify the class of the template, can extend the template by inheriting this class, see the chapter on custom templates for detailsdesc_template:Type[DescTemplate]=DescTemplate# Specify the protobuf files of which packages to ignore, and the messages of the ignored packages will not be parsedignore_pkg_list:List[str]=["validate","p2p_validate"]# Specifies the generated file name suffix (without .py)file_name_suffix="_p2p"Next, in order to be able to read this file, need to change the--protobuf-to-pydantic_out=.to--protobuf-to-pydantic_out=config_path=plugin_config.py:..
where the left side of:indicates that the configuration file path to be read isplugin_config.py, and the right side of:declares that the output location of theprotobuf-to-pydanticplugin is.The final complete command is as follows：python-mgrpc_tools.protoc-I.--protobuf-to-pydantic_out=config_path=plugin_config.py:.example.protoThrough this command, can load the corresponding configuration and run theprotobuf-to-pydanticplug-in。In addition to the configuration options in the example configuration file,
theprotobuf-to-pydanticplug-in also supports other configuration options.
The specific configuration instructions are as follows：Configuration nameFunctional moduleTypeHidden meaninglocal_dictTemplatedictHolds variables for thelocaltemplatedesc_templateTemplateprotobuf_to_pydantic.desc_template.DescTemplateImplementation of the template classcomment_prefixTemplatestrComment prefix.Only strings with a fixed prefix will be used by the templatecustomer_import_setCode generationSet[str]A collection of custom import statements, such asfrom typing import Setorimport typing, that will write data in order to the source code filecustomer_dequeCode generationdeque[str]Custom source file content, used to add custom contentmodule_pathstrstrUsed to define the root path of the project or module, which helpsprotobuf-to-pydanticto better automatically generate module import statementspyproject_file_pathCode generationstrDefine the pyproject file path, which defaults to the current project pathcode_indentCode generationintDefines the number of indentation Spaces in the code; the default is 4ignore_pkg_listCode generation(Limit plug-ins only)list[str]Definition ignores parsing of the specified package filebase_model_classModel Code generation, Code generationType[BaseModel]Define the parent class of the generated Modelfile_name_suffixCode generationstrDefine the generated file suffix, default_p2p.pyfile_descriptor_proto_to_codeCode generation(Limit plug-ins only)Type[FileDescriptorProtoToCode]Define theFileDescriptorProtoToCodeto use1.1.3.buf-cliIf you are usingbuf-clito manage Protobuf files,
then you can also useprotobuf-to-pydanticinbuf-cli, SeeHow to useprotobuf-to-pydanticinbuf-cli1.2.Generate aPydantic Modelobject in Python runtimeprotobuf_to_pydanticcan generate the correspondingPydanticModelobject based on theMessageobject at runtime。For example, theUserMessagein the following Protobuf file nameddemo.proto:// path: ./demo.protosyntax="proto3";packageuser;enumSexType{man=0;women=1;}messageUserMessage{stringuid=1;int32age=2;floatheight=3;SexTypesex=4;boolis_adult=5;stringuser_name=6;}grpc_tools.protoccan be used to generate the Python code file corresponding to theProtobuffile (the file name isdemo_pb2.py), and the code related to theUserMessageis stored in the code file.AtPythonruntime, The funcmsg_to_pydantic_modelcan be called to read theUserMessageobject from thedemo_pb2module and generate the correspondingPydantic Modelobject as follows:fromtypingimportTypefromprotobuf_to_pydanticimportmsg_to_pydantic_modelfrompydanticimportBaseModel# import protobuf gen python objfrom.importdemo_pb2UserModel:Type[BaseModel]=msg_to_pydantic_model(demo_pb2.UserMessage)print({k:v.field_infofork,vinUserModel.__fields__.items()})# output# {#   `uid`: FieldInfo(default=``, extra={}),#   `age`: FieldInfo(default=0, extra={}),#   `height`: FieldInfo(default=0.0, extra={}),#   `sex`: FieldInfo(default=0, extra={}),#   `is_adult`: FieldInfo(default=False, extra={}),#   `user_name`: FieldInfo(default=``, extra={})#  }Through the output results, it can be found that the generatedpydantic.BaseModelobject also containsuid,age,height,sex,is adultanduser namefields,
and thedefaultproperty matches the zero value of the Protobuf type。Themsg_to_pydantic_modelfunc is customizable just like plugins, with the following extension parameters:FieldsMeaningdefault_fieldGenerate aFieldfor each field in thePydantic Modelcomment_prefixThe prefix of a comment that can be parsedparse_msg_desc_methodParsing rules to uselocal_dictVariables used by thelocaltemplatepydantic_baseGenerates the parent class of thePydantic Modelobjectpydantic_moduleGenerate theModuleof thePydantic Modelobjectdesc_templateTemplate class to usemessage_type_dict_by_type_nameProtobuf type mapping toPythontypemessage_default_factory_dict_by_type_nameProtobuf type mapping to the Python type factoryIn addition to generating the correspondingPydantic Modelobject at runtime,protobuf-to-pydanticalso supports convertingPydantic Modelobjects to Python code text at runtime (only compatible withPydantic Modelobjects generated byprotobuf-to-pydantic).
Thepydantic_model_to_py_codefunc is used to generate the source code, and thepydantic_model_to_py_filefunc is used to generate the code file. The example code of thepydantic_model_to_py_filefunc is as follows:fromprotobuf_to_pydanticimportmsg_to_pydantic_model,pydantic_model_to_py_file# import protobuf gen python objfromexample.example_proto_python_code.example_proto.demoimportdemo_pb2pydantic_model_to_py_file("./demo_gen_code.py",msg_to_pydantic_model(demo_pb2.NestedMessage),)When the code runs, it convertsdemo_pb2.NestedMessageto a Pydantic Model object and passes it to thepydantic_model_to_py_file.pydantic_model_to_py_filegenerates the source code and writes it to ademo_gen_code.pyfile.2.Parameter validationIn the previous section, thePydantic Modelobject generated by the Protobuf file is very simple because the Protobuf file does not have enough parameters to verify the relevant information.
In order for each field in the generatedPydantic Modelobject to have parameter validation capabilities, the corresponding parameter checking rules for the field need to be refined in the Protobuf file.Currently,protobuf-to-pydanticsupports three validation rules：1.Text annotations2.PGV(protoc-geb-validate)3.P2PWith these rules, thePydantic Modelobject generated byprotobuf-to-pydanticwill have parameter validation feature.
Among them, text annotations and P2P rules are consistent, they both support most of the parameters inPydantic Field, some of the variations and new parameters are seen2.4.P2Pand text annotation rule other parameter supportNOTE:1.Text annotation rules are not the focus of subsequent functional iterative development, and it is recommended to use P2P verification rules.2.Protoc Plug-inonly supportPGVandP2Prule.2.1.Text annotationsIn the Protobuf file, can write annotations for each field that meet the requirements ofprotobuf-to-pydantic,
so thatprotobuf-to-pydanticcan obtain the validation information of the parameters when parsing the Protobuf file, such as the following examplesyntax="proto3";packageuser;enumSexType{man=0;women=1;}// user infomessageUserMessage{// p2p: {"required": true, "example": "10086"}// p2p: {"title": "UID"}stringuid=1;// p2p: {"description": "user union id"}// p2p: {"example": 18, "title": "use age", "ge": 0}int32age=2;// p2p: {"ge": 0, "le": 2.5}floatheight=3;SexTypesex=4;boolis_adult=5;// p2p: {"description": "user name"}// p2p: {"default": "", "min_length": 1, "max_length": "10", "example": "so1n"}stringuser_name=6;}In this example, each annotation that can be used byprotobuf_to_pydanticstarts withp2p:(supports customization) and is followed by a complete Json string. If are familiar with the usage ofpydantic, can find This Json string contains the verification information corresponding topydantic.Field. For example, theuidfield inUserMessagecontains a total of 4 pieces of information as follows：ColumnMeaningrequiredIndicates that the generated field does not have a default valueexampleAn example value representing the generated field is 10086titleIndicates that the schema name of the field is UIDdescriptionThe schema documentation for the representation field is described asuser_union_idNote:1.Currently only single-line comments are supported and comments must be a complete Json data (no line breaks).2.multi line comments are not supported。When these annotations are written,protobuf_to_pydanticwill bring the corresponding information for each field when converting the Message into the correspondingPydantic.BaseModelobject, as follows:# pydantic version V1fromtypingimportTypefromprotobuf_to_pydanticimportmsg_to_pydantic_modelfrompydanticimportBaseModel# import protobuf gen python objfromexample.example_proto_python_code.example_proto.demoimportdemo_pb2UserModel:Type[BaseModel]=msg_to_pydantic_model(demo_pb2.UserMessage,parse_msg_desc_method=demo_pb2)print({k:v.field_infofork,vinUserModel.__fields__.items()})# output# {#   `uid`: FieldInfo(default=PydanticUndefined, title=`UID`, description=`user union id`, extra={`example`: `10086`}),#   `age`: FieldInfo(default=0, title=`use age`, ge=0, extra={`example`: 18}),#   `height`: FieldInfo(default=0.0, ge=0, le=2, extra={}),#   `sex`: FieldInfo(default=0, extra={}),#   `is_adult`: FieldInfo(default=False, extra={}),#   `user_name`: FieldInfo(default=``, description=`user name`, min_length=1, max_length=10, extra={`example`: `so1n`})# }It can be seen that the output fields carry the corresponding information, which is consistent with the comments of the Protobuf file.In addition, this code differs from the previous section in that themsg_to_pydantic_modelfunction has a keyword argument namedparse_msg_desc_methodand its value is the 'demo_pb2' module.
This parameter enablesprotobuf-to-pydanticto obtain additional information about each field in the Message object through comments in the.pyifile of thedemo_pb2module.Note: This function needs to use themypy-protobufplugin when generating the correspondingPython code through the Protobuf file, and the specified pyi file output path is the same as the generatedPythoncode path to take effect. And need to installprotobuf-to-pydanticvia thepython -m pip install protobuf-to-pydantic[mypy-protobuf]` commandIn addition to getting comments from the.pyifile,protobuf-to-pydanticalso supports getting comment information for each field through comments on the Protobuf file to which the Message object belongs.
Using this feature is as simple as setting the value ofparse_msg_desc_methodto the root directory path specified when the Message object was generated.When using this method, make sure to installprotobuf-to-pydanticviapython -m pip install protobuf-to-pydantic[lark], and also make sure that the Protobuf file exists in the project.For example, the project structure of theprotobuf-to-pydanticsample code is as follows:./protobuf_to_pydantic/
├──example/
│├──python_example_proto_code/
│└──example_proto/
├──protobuf_to_pydantic/
└──/The Protobuf file is stored in theexample/example_protofolder, and then run the following command in theexampledirectory to generate thePythoncode file corresponding to Protobuf:cdexample

python-mgrpc_tools.protoc--python_out=./python_example_proto_code\--grpc_python_out=./python_example_proto_code\-I.\Then the path that needs to be filled in forparse_msg_desc_methodat this time is./protobuf_to_pydantic/example.
The following sample code:# pydantic Version v1fromtypingimportTypefromprotobuf_to_pydanticimportmsg_to_pydantic_modelfrompydanticimportBaseModel# import protobuf gen python objfromexample.proto_3_20_pydanticv1.example.example_proto.demoimportdemo_pb2UserModel:Type[BaseModel]=msg_to_pydantic_model(demo_pb2.UserMessage,parse_msg_desc_method="./protobuf_to_pydantic/example")print({k:v.field_infofork,vinUserModel.__fields__.items()})# output# {#   `uid`: FieldInfo(default=PydanticUndefined, title=`UID`, description=`user union id`, extra={`example`: `10086`}),#   `age`: FieldInfo(default=0, title=`use age`, ge=0, extra={`example`: 18}),#   `height`: FieldInfo(default=0.0, ge=0, le=2, extra={}),#   `sex`: FieldInfo(default=0, extra={}),#   `is_adult`: FieldInfo(default=False, extra={}),#   `user_name`: FieldInfo(default=``, description=`user name`, min_length=1, max_length=10, extra={`example`: `so1n`})# }As you can see, the only difference in this code is the value of theparse_msg_desc_method, but through the output result, you can see that the field carries the same information as the result obtained through the module.2.2.PGV(protoc-gen-validate)At present, the commonly used parameter verification project in the Protobuf ecosystem isprotoc-gen-validate，
It has become a common standard in Protobuf because it supports multiple languages and requires only one writing ofPGVrules to make the generatedMessageobject support the corresponding validation rules.Currentlyprotobuf-to-pydanticonly supports rules thatprotoc-gen-validateis less than version 1.0.0protobuf-to-pydanticsupports parsing ofPGVvalidation rules and generatesPydantic Modelobjects with validation logic functions.
UsingPGVchecksum rules inprotobuf-to-pydanticis very simple, just write the correspondingPGVrules in the Protobuf file first,
and then specify the value ofparse_msg_desc_methodto bePGVwhen callingmsg_to_pydantic_modelas the code below:# pydantic version V1fromtypingimportTypefromprotobuf_to_pydanticimportmsg_to_pydantic_modelfrompydanticimportBaseModel# import protobuf gen python objfromexample.proto_3_20_pydanticv1.example.example_proto.validateimportdemo_pb2UserModel:Type[BaseModel]=msg_to_pydantic_model(demo_pb2.FloatTest,parse_msg_desc_method="PGV")print({k:v.field_infofork,vinUserModel.__fields__.items()})# output# {#   `const_test`: FieldInfo(default=1.0, const=True, extra={}),#   `range_e_test`: FieldInfo(default=0.0, ge=1, le=10, extra={}),#   `range_test`: FieldInfo(default=0.0, gt=1, lt=10, extra={}),#   `in_test`: FieldInfo(default=0.0, extra={`in`: [1.0, 2.0, 3.0]}),#   `not_in_test`: FieldInfo(default=0.0, extra={`not_in`: [1.0, 2.0, 3.0]}),#   `ignore_test`: FieldInfo(default=0.0, extra={})# }Note:1.For the usage ofPGV, see:protoc-gen-validate doc2.Need to installPGVthroughpip install protoc_gen_validateor downloadvalidate.prototo the protobuf directory in the project to write pgv rules in the Protobuf file.2.3.P2PThePGVverification rules are written in the Option attribute of each field ofMessageand have a better code specification,
so Protobuf that usePGVchecksum rules will be more readable than Protobuf that use annotation .At the same time, when writingPGVrules, can also experience the convenience of the IDE's auto-completion and the security of checksumming when generating the corresponding language objects from Protobuf files, but it only supports checksumming-related logic, which is not as rich as the file annotation mode.TheP2Pverification rule that comes withprotobuf-to-pydanticexpands on thePGVverification rule by incorporating some of the functionality of the text annotation verification rule, which satisfies most of the customization of the properties of eachFieldin thePydantic Model, such as the following Protobuf file.syntax="proto3";packagep2p_validate_test;import"example_proto/common/p2p_validate.proto";messageFloatTest{floatconst_test=1[(p2p_validate.rules).float.const=1];floatrange_e_test=2[(p2p_validate.rules).float={ge:1,le:10}];floatrange_test=3[(p2p_validate.rules).float={gt:1,lt:10}];floatin_test=4[(p2p_validate.rules).float={in:[1,2,3]}];floatnot_in_test=5[(p2p_validate.rules).float={not_in:[1,2,3]}];floatdefault_test=6[(p2p_validate.rules).float.default=1.0];floatnot_enable_test=7[(p2p_validate.rules).float.enable=false];floatdefault_factory_test=8[(p2p_validate.rules).float.default_factory="p2p@builtin|float"];floatmiss_default_test=9[(p2p_validate.rules).float.miss_default=true];floatalias_test=10[(p2p_validate.rules).float.alias="alias"];floatdesc_test=11[(p2p_validate.rules).float.description="test desc"];floatmultiple_of_test=12[(p2p_validate.rules).float.multiple_of=3.0];floatexample_test=13[(p2p_validate.rules).float.example=1.0];floatexample_factory=14[(p2p_validate.rules).float.example_factory="p2p@builtin|float"];floatfield_test=15[(p2p_validate.rules).float.field="p2p@local|CustomerField"];floattype_test=16[(p2p_validate.rules).float.type="p2p@local|confloat"];floattitle_test=17[(p2p_validate.rules).float.title="title_test"];}protobuf-to-pydanticcan read the generated Message object at runtime and generate aPydantic Modelobject with the corresponding information:# pydantic version V1fromtypingimportTypefromprotobuf_to_pydanticimportmsg_to_pydantic_modelfrompydanticimportBaseModel,confloatfrompydantic.fieldsimportFieldInfo# import protobuf gen python objfromexample.proto_3_20_pydanticv1.example.example_proto.p2p_validateimportdemo_pb2classCustomerField(FieldInfo):passDemoModel:Type[BaseModel]=msg_to_pydantic_model(demo_pb2.FloatTest,local_dict={"CustomerField":CustomerField,"confloat":confloat},)print({k:v.field_infofork,vinDemoModel.__fields__.items()})# output:# {#   'const_test': FieldInfo(default=1.0, const=True, extra={}),#   'range_e_test': FieldInfo(default=0.0, ge=1, le=10, extra={}),#   'range_test': FieldInfo(default=0.0, gt=1, lt=10, extra={}),#   'in_test': FieldInfo(default=0.0, extra={'in': [1.0, 2.0, 3.0]}),#   'not_in_test': FieldInfo(default=0.0, extra={'not_in': [1.0, 2.0, 3.0]}),#   'default_test': FieldInfo(default=1.0, extra={}),#   'default_factory_test': FieldInfo(default=PydanticUndefined, default_factory=<class 'float'>, extra={}),#   'miss_default_test': FieldInfo(extra={}),#   'alias_test': FieldInfo(default=0.0, alias='alias', alias_priority=2, extra={}),#   'desc_test': FieldInfo(default=0.0, description='test desc', extra={}),#   'multiple_of_test': FieldInfo(default=0.0, multiple_of=3, extra={}),#   'example_test': FieldInfo(default=0.0, extra={'example': 1.0}),#   'example_factory': FieldInfo(default=0.0, extra={'example': <class 'float'>}),#   'field_test': CustomerField(default=0.0, extra={}),#   'type_test': FieldInfo(default=0.0, extra={}),#   'title_test': FieldInfo(default=0.0, title='title_test', extra={})#   }Note:1.See the2.5.templatefor the usage oflocal_dict2.If the reference to the Proto file fails, need to downloadp2p_validate.protoin the project and use it in the Protobuf file。2.4.P2Pand text annotation rule other parameter supportTheprotobuf-to-pydantictext annotation rules and theP2Prules support most of the parameters inFieldInfo, as described in thePydantic Field doc。The new parameters added toPydantic V2will be supported in version 2.1, for nowP2Prule naming is still written on the basis ofPydantic V1, but automatic mapping toPydantic V2naming is supported.Other partial changes in meaning and new parameters are described as follows:ParameterDefault valueIllustraterequiredFalseBy default, the default value of each field in the generatedPydantic Modelobject is the same as the zero value of its corresponding type. WhenrequiredisTrue, no more default values are generated for the fields.enableTrueBy default,protobuf-to-pydanticgenerates all fields forMessage, if don't want the generatedMessageto have this field, can setenabletoFalse.constNoneUsed to specify a constant value for a field, though differentPydanticversions behave differentlyForPydantic V1, the value ofdefaultinFieldis set to the value specified byconst, andconstinFieldis set to True.Note:Pydantic Model's const only supports bool variables, whenconstisTrue, the accepted value can only be the value set bydefault, and the default value carried by the message generated by protobuf is the zero value of the corresponding type does not match withPydantic Model, soprotobuf-to-pydanticmakes some changes to the input of this value.ForPydantic V2, the value ofdefaultinFieldremains the same, but the type annotation changes totyping_extensions.Literal[xxx]typeNoneBy default, the default type of a field is the same as Protobuf's, but use the2.5.templatefunction to modify the type of a field.extraNoneTheextraparameter accepted byPydanticis of typePython Dict, which is not supported by Protobuf, and requires the use of either2.5.Templatesor the corresponding Json structureprotobuf-to-pydanticin the Protobuf file to parse it properly.fieldNoneBy default, theFieldof the parameter isPydantic FieldInfo, although it can be customized using the2.5.Templatesfunctiondefault_templateNoneSimilar todefault, default values can be customized in fields that are not of string type using the2.5.Templatesfeature.In addition to the above parameters, also support for fast import ofPydantic typefor string types. For example, if want to add a check for card numbers via thepydantic.types.PaymentCardNumbertype, can specify the type of thepydantic_typeparameter field to bePaymentCardNumber, which is similar to the use of template imports in thetyperule, as follows:Text annotation rules：syntax="proto3";packagecommon_validate_test;// common examplemessageUserPayMessage{stringbank_number=1;// p2p: {"pydantic_type": "PaymentCardNumber"}stringother_bank_number=2;// p2p: {"type": "p2p@import|pydantic.types|PaymentCardNumber"}}P2P rules：syntax="proto3";packagep2p_validate_test;import"example_proto/common/p2p_validate.proto";// p2p examplemessageUserPayMessage{stringbank_number=1[(p2p_validate.rules).string.pydantic_type="PaymentCardNumber"];stringother_bank_number=2[(p2p_validate.rules).string.type="p2p@import|pydantic.types|PaymentCardNumber"];}SeeExtra Types Overviewfor supported `Pydantic Types'.2.5.TemplateWhen working with definition fields, will find that some fields are filled with values that are methods or functions of one of the libraries inPython(e.g., the values of thetypeparameter and thedefault_factoryparameter), which can't be accomplished with the Json syntax.
At this point, templates can be used to solve the corresponding problem, and currentlyprotobuf-to-pydanticsupports a variety of template functiNote: Thep2pstring at the beginning of a template can be defined via the comment_prefix variable2.5.1.p2p@importTemplateThep2p@importtemplate is used to represent variables in other modules that need to be introduced before they can be used, as follows.Examples of text annotation rules：syntax="proto3";packagecomment_validate_test;// comment examplemessageUserPayMessage{stringbank_number=1;// p2p: {"type": "p2p@import|pydantic.types|PaymentCardNumber"}}Examples of P2P rules (1)：syntax="proto3";packagep2p_validate_test;import"example_proto/common/p2p_validate.proto";messageUserPayMessage{stringbank_number=1[(p2p_validate.rules).string.type="p2p@import|pydantic.types|PaymentCardNumber"];}Examples of P2P rules (2)：syntax="proto3";packagep2p_other_validate_test;import"example_proto/common/p2p_validate.proto";// p2p other examplemessageUserPayMessage{stringbank_number=1[(p2p_validate.rules).string.pydantic_type="PaymentCardNumber"];}The example Protobuf file uses a syntax in the formatp2p@{methods of the template}|{modules to be imported:A}|{variables in modules:B}, indicating that aBobject needs to be imported byfrom A import Band used by the corresponding rule.
With the definition of the template,protobuf-to-pydanticconverts the corresponding Message into aPydantic Model, as follows:frompydanticimportBaseModelfrompydantic.fieldsimportFieldInfo# p2p@import|pydantic.types|PaymentCardNumberfrompydantic.typesimportPaymentCardNumberclassUserPayMessage(BaseModel):bank_number:PaymentCardNumber=FieldInfo(default="",extra={})2.5.2.p2p@import_instanceTemplateThep2p@import_instancetemplate introduces the class of a library and then instantiates it in combination with the specified parameters before it is used by the corresponding rule, which is used as follows:syntax="proto3";packagep2p_validate_test;import"google/protobuf/any.proto";import"example_proto/common/p2p_validate.proto";// p2p examplemessageAnyTest{google.protobuf.Anydefault_test=23[(p2p_validate.rules).any.default='p2p@import_instance|google.protobuf.any_pb2|Any|{"type_url": "type.googleapis.com/google.protobuf.Duration"}'];}The syntax used here isp2p@{methods of the template}|{modules to be introduced}|{classes to be introduced}|{initialization parameters}, and the definition ofprotobuf-to-pydanticthrough the template will turn the corresponding Message into the followingPydantic Modelobject:fromgoogle.protobuf.any_pb2importAnyasAnyMessagefrompydanticimportBaseModelfrompydantic.fieldsimportFieldInfoclassAnyTest(BaseModel):default_test:AnyMessage=FieldInfo(default=AnyMessage(type_url="type.googleapis.com/google.protobuf.Duration"))2.5.3.p2p@localTemplateThis template is used to introduce user-defined variables, using a syntax in the format{method of the template}|{local variable to be used}, as follows:Example of text annotation:syntax="proto3";packagecomment_validate_test;import"google/protobuf/timestamp.proto";import"example_proto/common/p2p_validate.proto";// comment examplemessageUserPayMessage{google.protobuf.Timestampexp=1;// p2p: {"default_factory": "p2p@local|exp_time"}}Examples of P2P rules：syntax="proto3";packagep2p_validate_test;import"google/protobuf/timestamp.proto";import"example_proto/common/p2p_validate.proto";// p2p examplemessageUserPayMessage{google.protobuf.Timestampexp=1[(p2p_validate.rules).timestamp.default_factory="p2p@local|exp_time"];}However, themsg_to_pydantic_modelfunc needs to be called with the parameterlocal_dictto register the corresponding value, the pseudo-code is as follows:# a.pyimporttimefromexample.proto_3_20_pydanticv1.example.example_proto.p2p_validateimportdemo_pb2fromprotobuf_to_pydanticimportmsg_to_pydantic_modeldefexp_time()->float:returntime.time()msg_to_pydantic_model(demo_pb2.NestedMessage,local_dict={"exp_time":exp_time},# <----  use local_dict)In this way,protobuf-to-pydanticgenerates a conformingPydantic Modelobject:# b.pyfromdatetimeimportdatetimefrompydanticimportBaseModelfrompydantic.fieldsimportFieldInfofromaimportexp_time# <-- exp_time in a.pyclassUserPayMessage(BaseModel):exp:datetime=FieldInfo(default_factory=exp_time,extra={})2.5.4.p2p@builtinTemplateThis template (which can be thought of as a simplified version of thep2p@localtemplate) can be used directly when the variables to be used come fromPythonbuilt-in functions,the syntax is used as follows:Examples of text annotation rules:syntax="proto3";packagecomment_validate_test;import"google/protobuf/timestamp.proto";import"example_proto/common/p2p_validate.proto";// comment examplemessageUserPayMessage{google.protobuf.Timestampexp=1;// p2p: {"type": "p2p@builtin|float"}}Examples of P2P rules：syntax="proto3";packagep2p_validate_test;import"google/protobuf/timestamp.proto";import"example_proto/common/p2p_validate.proto";// p2p examplemessageUserPayMessage{google.protobuf.Timestampexp=1[(p2p_validate.rules).timestamp.type="p2p@builtin|float"];}Then can directly generate a conformingPydantic Modelobject by calling themsg_to_pydantic_modelfunction, as follows:frompydanticimportBaseModelfrompydantic.fieldsimportFieldInfoclassUserPayMessage(BaseModel):exp:float=FieldInfo()2.5.5.Customized templatesCurrentlyprotobuf-to-pydanticonly supports a few simple templates, if have more template needs, can extend the templates by inheriting theDescTemplateclass.For example, there is an odd feature that requires the default value of a field to be the timestamp of the time when thePydantic Modelobject was generated, but the timestamps used are available in lengths of 10 and 13, so the following Protobuf file needs to be written to support defining the length of the timestamps:syntax="proto3";packagep2p_validate_test;import"google/protobuf/timestamp.proto";import"example_proto/common/p2p_validate.proto";messageTimestampTest{int32timestamp_10=1[(p2p_validate.rules).int32.default_template="p2p@timestamp|10"];int32timestamp_13=2[(p2p_validate.rules).int32.default_template="p2p@timestamp|13"];}As you can see, the Protobuf file customizes the syntax ofp2p@timestamp|{x}, wherexhas only two values, 10 and 13. The next step is to write code based on this template behavior, which looks like this.importtimefromprotobuf_to_pydantic.gen_modelimportDescTemplateclassCustomDescTemplate(DescTemplate):deftemplate_timestamp(self,length_str:str)->int:timestamp:float=time.time()iflength_str=="10":returnint(timestamp)eliflength_str=="13":returnint(timestamp*100)else:raiseKeyError(f"timestamp template not support value:{length_str}")from.demo_pb2importTimestampTest# fake codefromprotobuf_to_pydanticimportmsg_to_pydantic_modelmsg_to_pydantic_model(TimestampTest,desc_template=CustomDescTemplate# <-- Use a custom template class)This code first creates a classCustomDescTemplatethat inherits fromDescTemplate.DescTemplatewill forwards to the correspondingtemplate_{template name}method based on the naming of the template, so this class needs to define thetemplate_timestampmethod to implement thep2p@timestamptemplate functionality.
In addition, thelength_strvariable received in this method is either 10 inp2p@timestamp|10or 13 inp2p@timestamp|13.Then load theCustomDescTemplatethrough themsg_to_pydantic_modelfunction, then the following code will be generated (assuming that the code is generated at a timestamp of 1600000000):frompydanticimportBaseModelfrompydantic.fieldsimportFieldInfoclassTimestampTest(BaseModel):timestamp_10:int=FieldInfo(default=1600000000)timestamp_13:int=FieldInfo(default=1600000000000)3.Code formatThe code generated directly throughprotobuf-to-pydanticis not perfect, but it is possible to indirectly generate code that conforms to thePythonspecification through different formatting tools.
Currently,protobuf-to-pydanticsupports formatting tools such asautoflake,blackandisort. If the corresponding formatting tool is installed in the currentPythonenvironment, thenprotobuf-to-pydanticwill call the tool to format the generated code before outputting it to a file.In addition, the decision to enable or disable a formatting tool can be made through thepyproject.tomlconfiguration file, thepyproject.tomlexample of which reads as follows:# Controls which formatting tools protobuf-to-pydantic uses,# if false then no formatting tools are used (default is true)[tool.protobuf-to-pydantic.format]black=trueisort=trueautoflake=true# black docc:https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#configuration-format[tool.black]line-length=120target-version=['py37']# isort doc:https://pycqa.github.io/isort/docs/configuration/config_files.html#pyprojecttoml-preferred-format[tool.isort]profile="black"multi_line_output=3include_trailing_comma=trueforce_grid_wrap=0use_parentheses=trueensure_newline_before_comments=trueline_length=120# autoflake doc:https://github.com/PyCQA/autoflake#configuration[tool.autoflake]in-place=trueremove-all-unused-imports=trueremove-unused-variables=true4.exampleprotobuf-to-pydanticprovides some simple example code for reference only.4.1.Generate code directlyProtobuf file:demo/demo.protoGeneratePydantic Model(Pydantic V1):proto_pydanticv1/demo_gen_code.pyGeneratePydantic Model(Pydantic V2):proto_pydanticv2/demo_gen_code.py4.2.Text annotationProtobuf File:demo/demo.protoPydantic Modelgenerated based onpyifile(Pydantic V1):proto_pydanticv1/demo_gen_code_by_text_comment_pyi.pyPydantic Modelgenerated based onpyifile(Pydantic V2):proto_pydanticv2/demo_gen_code_by_text_comment_pyi.pyPydantic Modelgenerated based on protobuf file(Pydantic V1):proto_pydanticv1/demo_gen_code_by_text_comment_protobuf_field.pyPydantic Modelgenerated based on protobuf file(Pydantic V2):proto_pydanticv2/demo_gen_code_by_text_comment_protobuf_field.pyvalidate/demo.proto](https://github.com/so1n/protobuf_to_pydantic/blob/master/example/example_proto/validate/demo.proto)GeneratePydantic Model(Pydantic V1):proto_pydanticv1/demo_gen_code_by_pgv.pyGeneratePydantic Model(Pydantic V2):proto_pydanticv2/demo_gen_code_by_pgv.py4.4.P2P ruleProtobuf file:p2p_validate/demo.protoGeneratePydantic Model(Pydantic V1):proto_pydanticv1/demo_gen_code_by_p2p.pyGeneratePydantic Model(Pydantic V2):proto_pydanticv2/demo_gen_code_by_p2p.py4.5.Protoc Plugin-inProtobuf field:demo/demo.proto，validate/demo.proto，p2p_validate/demo.protoNote: The Protoc plugin only supports P2P and PGV rulesPydantic Modelgenerated viademo/demo.proto(Pydantic V1):example_proto/demo/demo_p2p.pyPydantic Modelgenerated viademo/demo.proto(Pydantic V2):example_proto/demo/demo_p2p.pyPydantic Modelgenerated viavalidate/demo.proto(Pydantic V1):example_proto/validate/demo_p2p.pyPydantic Modelgenerated viavalidate/demo.proto(Pydantic V1):example_proto/validate/demo_p2p.pyPydantic Modelgenerated viap2p_validate/demo.proto(Pydantic V1):example_proto/p2p_validate/demo_p2p.pyPydantic Modelgenerated viap2p_validate/demo.proto(Pydantic V1):example_proto/p2p_validate/demo_p2p.py