Project: gRPC (Orchestration & Management / Remote Procedure Call)
File: implementation_notes.md

This is an overview for maintainers of Grpc.Tools. The Grpc.Tools NuGet package provides custom build targets to make it easier to specify `.proto` files in a project and for those files to be compiled and their generated files to be included in the project. MSBuild properties and targets included from the Grpc.Tools NuGet package are in: `build\Grpc.Tools.props`, which imports `build\grpc\Grpc.Tools.props` `build\_protobuf\Google.Protobuf.Tools.props` `build\Grpc.Tools.targets`, which imports `build\grpc\Grpc.Tools.targets` `build\_protobuf\Google.Protobuf.Tools.targets` Details of how NuGet packages can add custom build targets and properties to a project is documented here: Basically the `.props` and `.targets` files are automatically included in the projects - the `.props` at the top of the project and the `.targets` are added to the bottom of the project. For Visual Studio integration - these files provide the properties pages: `build\_protobuf\Protobuf.CSharp.xml` (included from `Google.Protobuf.Tools.targets`) `build\grpc\Grpc.CSharp.xml` (included from `Grpc.Tools.targets`) DLLs containing the custom tasks are in: `build\_protobuf\netstandard1.3` `build\_protobuf\net45` Native binary executables for the protobuf compiler (protoc) and C# gRPC plugin (grpccsharpplugin) are included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and CPU architectures (x86, x64, arm64). The build determines which executables to use for the particular machine that the it is being run on. These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables: `ProtobufProtocFullPath` property or `PROTOBUFPROTOC` environment variable \ Full path of protoc executable `gRPCPluginFullPath` property or `GRPCPROTOC_PLUGIN` environment variable \ Full path of gRPC C# plugin The custom targets hook into various places in a normal MSBuild build by specifying before/after targets at the relevant places. See for predefined targets. Before `PrepareForBuild` we add `Protobuf_SanityCheck` that checks this is a supported project type, e.g. a C# project. Before `BeforeCompile` we add all the targets that compile the `.proto` files and generate the expected `.cs` files. These files are added to those that get compiled by the C# compiler. The target `ProtobufCompileBeforeCsCompile` is the glue_ inserting the targets into the build. It may look like it isn't doing anything but by specifying `BeforeTargets` and `DependsOnTargets` it inserts `Protobuf_Compile` into the build - but only doing so if this is a C# project. After `CoreClean` we add `Protobuf_Clean` that cleans the files generated by the protobuf compiler. The target `ProtobufCleanAfterCsClean` is the glue inserting `ProtobufClean` into the build - but only doing so if this is a C# project. There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project and packaged in the file `Protobuf.MSBuild.dll` in the NuGet package. See for information about implementing custom tasks. ProtoToolsPlatform Works out the operating system and CPU architecture ProtoCompilerOutputs Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items Also returns list of items that is the same as the `Protobuf` items passed in with the output directory metadata updated ProtoReadDependencies Read generated files from previously written dependencies file and return as

ProtoCompile Runs the protobuf compiler for a proto file. The executable to run is specified by the property `Protobuf_ProtocFullPath` To do this it: first writes out a response file containing the parameters for protobuf compiler runs the executable, which generates the `.cs` files and a `.protodep` dependencies file reads the dependencies file to find the files that were generated and these are returned as a list of items that can then be used in the MSBuild targets The names of these items and properties are correct at the time of writing this document. High level builds steps: Prepare list of `.proto` files to compile Makes sure all needed metadata is set for the `<Protobuf>` item, defaulting some values Removes `<Protobuf>` items that no longer exist or are marked as don't compile Handling incremental builds Work out files that need to be created or have changed Compile the `.proto` files Add generated files to the list of files for the C# compiler At various stages of the build copies of the original `<Protobuf>` items are created and/or updated to set metadata and to prune out unwanted items. Firstly, the build makes sure `ProtoRoot` metadata is set for all `Protobuf` items. A new list of items - `Protobuf_Rooted` - is created from the `Protobuf` items with `ProtoRoot` metadata set: If `ProtoRoot` already set in the `<Protobuf>` item in the project file then it is left as-is. If the `.proto` file is under the project's directory then set `ProtoRoot="."`. If the `.proto` file is outside of the project's directory then set `ProtoRoot="<relative path to project directory>"`. Now prune out from `Protobuf_Rooted` the items that the user doesn't want to compile - those don't have `ProtoCompile` metadata as `true`. The pruned list is now called `Protobuf_Compile`. Set the `Source` metadata on `Protobuf_Compile` items to be the name of the `.proto` file. The `Source` metadata is used later as a key to map generated files to `.proto` files. The target `Protobuf_PrepareCompile` tries to work out which files the protobuf compiler will generate without actually calling the protobuf compiler. This is a best-effort guess. The custom task `ProtoCompilerOutputs` is called to do this. The results are stored in the item list `Protobuf_ExpectedOutputs`. The target `Protobuf_PrepareCompile` also reads previously written `.protodep` files to get any actual files previously generated. The custom task `ProtoReadDependencies` is called to do this. The results are stored in the item list `Protobuf_Dependencies`. This is in case the list of actual files is different from the previous best-effort guess from `ProtoCompilerOutputs`. The expected outputs and previous outputs are needed so that the timestamps of those files can be checked later when handling an incremental build. To avoid unnecessarily recompiling the `.proto` files during an incremental build the target `ProtobufGatherStaleBatched` tries to work out if any files have changed. It checks for out of date files using MSBuilds incremental build feature that compares the timestamps on a target's Input files to its Output

See The Inputs that are checked are: Timestamps of the `.proto` files Timestamps of previously generated files (list of these files read from `.protodep` files) Timestamps of MSBuild project files These are checked against the Outputs: Timestamps of the expected generated files is used to check each `.proto` file against its expected outputs. The batching is done by specifying the `Source` metadata in the Input. Items where `Source` metadata matches in both input and output are in each batch. The target `ProtobufGatherStaleBatched` sets the metadata `Exec=true` on `Protobuf_OutOfDateProto` items that are out of date. Later in the target `ProtobufGatherStaleFiles`, the items in `ProtobufOutOfDateProto` that don't have metadata `_Exec==true` are removed from the list of items, leaving only those that need compiling. The target `ProtobufCoreCompile` is run for each `.proto` file that needs compiling. These are in the item list `ProtobufOutOfDateProto`. The custom task `ProtoCompile` is called to run the protobuf compiler. The files that were generated are returned in the item list `ProtobufGeneratedFiles`. If there are expected files that were not actually generated then the behaviour depends on whether the generated files should have been within the project (e.g. in the intermediate directories) or were specified to be outside of the project. If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable) TODO: why are files inside and outside the project treated differently? The target `ProtobufAugmentLanguageCompile` adds to the `Compile` item list (the list of files that CSC compiles) the expected generated files. Note - this is the expected files not the actual generated files and this is done before the protobuf compiler is called. TODO: why are the expected files not the actual generated files added? Design-time builds are special builds that Visual Studio uses to gather information about the project. They are not user-initiated but may be triggered whenever files are added, removed or saved. See . The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the protobuf compiler during a design time build. However this optimisation can lead to errors in Visual Studio because the generated files may not exist or be out of date and any code that relies on them will then have errors. Now design time builds behave exactly the same as a normal build. The old behaviour can be enabled by setting the setting `DisableProtobufDesignTimeBuild` property to `true` in the project file _if_ it is a design time build, e.g. by adding ```xml <PropertyGroup Condition="'$(DesignTimeBuild)' == 'true' "> <DisableProtobufDesignTimeBuild>true</DisableProtobufDesignTimeBuild> </PropertyGroup> ``` For SDK projects it is possible to automatically include `.proto` files found in the project directory or sub-directories, without having to specify them with a `<Protobuf>` item. To do this the property `EnableDefaultProtobufItems` has be set to `true` in the project file. By default it is not set and `<Protobuf>` items must be included in the project for the `.proto` files to be compiled.