Package: protogen
protogenPackageprotogenmakes writingprotocplugins easier.
Working with the raw protobuf descriptor messages can be cumbersome.protogenresolves and links the dependencies and references between the raw Protobuf descriptors and turns them into their correspondingprotogenclasses that are easier to work with.
It also provides mechanisms that are espacially useful to generate Python code like dealing with Python imports.InstallationPackageprotogenis available viapip. To install run:pip install protogenAPIMost classes inprotogenare simply replacements of their corresponding Protobuf descriptors:protogen.Filerepresents a FileDescriptor,protogen.Messagea Descriptor,protogen.Fielda FieldDescriptor and so on. They should be self explanatory. You canread the docstringsfor more information about them.The classesprotogen.Options,protogen.Pluginandprotogen.GeneratedFilemake up a framework to generate files.
You can see these in action in the following example plugin:#!/usr/bin/env python"""An example plugin."""importprotogendefgenerate(gen:protogen.Plugin):forfingen.files_to_generate:g=gen.new_generated_file(f.proto.name.replace(".proto",".py"),f.py_import_path,)g.P("# Generated code ahead.")g.P()g.print_imports()g.P()forminf.message:g.P("class ",m.py_ident,":")forffinm.fields:# ...forsinf.services:g.P("class ",s.py_ident,":")forminf.methods:g.P("  def ",m.py_name,"(request):")g.P("    pass")if__name__=="__main__":opts=protogen.Options()opts.run(generate)classprotogen.OptionsTheprotogen.Optionsclass can be used to specify options for the resolution process (resolution from plain proto descriptors toprotogenclasses).Option.run(f: func(Plugin))waits forprotocto write the CodeGeneratorRequest tostdin, resolves the descriptors contained in it to their correspondingprotogenclasses and initializes a newPluginwith the resolved classes.fis then called with thePluginas argument.Oncefreturns,Optionswill collect the CodeGeneratorResponse from thePluginthat contains the all createdGeneratedFiles and write it tostdoutforprotocto pick it up.protocwrites the generated files to disk.classprotogen.PluginThePluginclass holds the files code generation is requested for in thePlugin.files_to_generateattribute. These are the files that were provided as command line arguments toprotoc.
Any options/parameters passed to the plugin via theprotoc --plugin_opt=<param>command line flag are accessible viaPlugin.parameter.
WithPlugin.new_generated_filea newGeneratedFilegets created that is automatically added to the CodeGeneratorResponse of the plugin.
Typically, but not necessarily, one file for each file inPlugin.files_to_generateis created.classprotogen.GeneratedFileTheGeneratedFileis just a buffer you can add lines to using theg.P(print) method.
AGeneratedFileis created withPlugin.new_generated_file(filename, py_import_path).
Thefilenameis obviously the name of the file to be created.
Thepy_import_pathis used forimport resolution.Note that the following assumes the plugin generates Python code. For other kinds of plugins, the following is not relevant:It is often necessary to import Python identifiers that are defined in different Python modules.
For example, a Protobuf messages might referencegoogle.protobuf.Timestampin one of its fields.
The corresponding Python classgoogle.protobuf.timestamp_pb2.Timestampneeds to be imported before its mentioned in the generated code.Theprotogen.PyImportPathclass represent a Python import path. Is just a wrapper around an import path (for example"google.protobuf.timestamp_pb2").
ThePyIdentclass represent a Python identifier. It holds aPyImportPathtogether with a name (e.g. a class name like"Timestamp").Theprotogen.GeneratedFileprovides mechanisms to handle Python imports.
Internally it maintains a list ofPyImportPaths that it needs to import.PyImportPathsmight be added to this list implictly when callingGeneratedFile.P(*args)or rather explicitly when callingGeneratedFile.qualified_py_ident(PyIdent).
When any of the arguments toGeneratedFile.Pis aprotogen.PyIdent, thepy_import_pathof theGeneratedFilegets compared to the argumentsPyIdent.py_import_path.If they are from different Python modules, the arguments import path will be added to the list of imports and the fully qualified name of thePyIdentwill be printed.If both files are from the samePyImportPath, then the import path is not added to the list of imports. In that case it is sufficient to reference thePyIdentby its simple name (e.g.Timestamp), thus only thePyIdent.py_namewill be printed.To place the import statements in the buffer of theGeneratedFileuseGeneratedFile.print_imports. This will put a line"import <path>"for eachPyImportPaththat the generated file needs to import (e.g"import google.protobuf.timestamp_pb2") in the buffer.The following example shows how theGeneratedFile.Pfunction behaves for differentPyImportPaths::# g is of type protogen.GeneratedFile# message_a and message_b are of type protogen.Message>>>g.py_import_path{"mypackage.mymodule"}>>>message_a.py_ident{py_import_path:"google.protobuf.timestamp_pb2",py_name:"Timestamp"}>>>g.P("hello ",message_a.py_ident)# adds "hello google.protobuf.timestamp_pb2.Timestamp" to g's line buffer and "google.protobuf.timestamp_pb2" to the imports>>>message_b.py_ident{py_import_path:"mypackage.mymodule",py_name:"MyMessage"}>>>g.P("hello ",message_b.py_ident)# adds "hello MyMessage" to g's line buffer (and nothing to the imports)Note that you can provide a custompy_import_funcin theOptionsconstructor.
This function is used in the resolution process to calculate thePyImportPathforprotogen.Files.protogen.Messages,protogen.Services andprotogen.Enums inherit thePyImportPath(that is part of theirPyIdent) from the file they are defined in.
By default theprotogen.default_py_import_funcis used.
It is compatible with the style of the offical Pythonprotocplugin that generates for each input filepath/to/file.protoa correspondingpath/to/file_pb2.pyfile.For example, assume you know that code generation for proto definitions that are part of themypackage.**proto package happens with aprotocplugin that generates one.pyfile per proto package.
Thatpluginalso omits the_pb2suffix.
For the proto packagemypackage.api.a, that might contain any number of files, it creates amypackage/api/a.pyfile.
For the proto packagemypackage.api.b, amypackage/api/b.pyfile.Apy_import_funcdescribing this would be:defpy_import_func(proto_filename:str,proto_package:str,)->protogen.PyImportPath:ifproto_package.split(".")[0]=="mypackage":# Python import path is simply the package name.returnprotogen.PyImportPath(proto_package)# For every other package, assume its generated with the offical Python plugin.returnprotogen.default_py_import_func(proto_filename,proto_package)MiscWhat is a protoc plugin anyway?protoc, theProtobufcompiler, is used to generate code derived from Protobuf definitions (.protofiles).
Under the hood,protoc's job is to read and parse the definitions into theirDescriptortypes (seegoogle/protobuf/descriptor.proto).
Whenprotocis run (with a plugin) it creates a CodeGeneratorRequest (seegoogle/protobuf/compiler/plugin.proto#L68) that contains the descriptors for the files to generate and everything they import and passes it to the plugin viastdin.Aprotoc pluginis an executable. It reads the CodeGeneratorRequest fromstdinand returns a CodeGeneratorResponse (seegoogle/protobuf/compiler/plugin.proto#L99) viastdout.
The plugin can use the descriptors from the CodeGeneratorRequest to create output files (in memory).
It returns these output files (consisting of name and content as string) in the CodeGeneratorResponse toprotoc.protocthen writes these files to disk.Runprotocwith your pluginAssume you have an executable plugin underpath/to/plugin/main.py.
You can invoke it via:protoc 
    --plugin=protoc-gen-myplugin=path/to/plugin/main.py \
    --myplugin_out=./output_root \
    myproto.proto myproto2.protoCaveats:you must use the--plugin=protoc-gen-<plugin_name>prefix, otherwiseprotocfails with "plugin not executable"specify the output path of the plugin with--<plugin_name>-outflag where<plugin_name>is the same as used in the--pluginflagyour plugin must be executable (chmod +x path/to/plugin/main.pyand put a#!/usr/bin/env pythonat the top of the file)See alsoif you want to write protoc plugins with JavaScript/TypeScript:github.com/fischor/protogen-javascriptif you want to write protoc plugins with Golang:google.golang.org/protobuf/compiler/protogenCreditsThis package is inspired by thegoogle.golang.org/protobuf/compiler/protogen Golangpackage.