Repository: bufbuild/buf
Pull request #915 (2022-02-04 16:58:47)

Delete `buf protoc`
This PR effectively deletes `buf protoc`.

`buf protoc` was added to demonstrate to users that `buf` actually does produce equivalent compilation results to `protoc`, and to give them a mechanism to verify this. However, `protoc` is a poor experience for working with Protobuf - this poor experience is part of why `buf` exists. The experience is so sufficiently different (and difficult to isolate) that we have what amounts to a very different internal code path to setup the compilation. These differences result in a maintenance burden for something that we want to deprecate in the first place.

Issues are piling up that we don't have bandwidth to address anytime soon:

- https://github.com/bufbuild/buf/issues/776
- https://github.com/bufbuild/buf/issues/491
- https://github.com/bufbuild/buf/issues/353
- https://github.com/bufbuild/buf/issues/351
- https://github.com/bufbuild/buf/issues/357

We want to spend our time fixing the Protobuf experience as a whole, not maintaining something that isn't a great experience in the first place. As such, we're removing `buf protoc` pre-v1.0.

What this PR actually does is moves `buf protoc` to `buf alpha protoc`. `buf alpha` is a hidden, completely experimental sub-command that has zero guarantees - in fact, we expect most of the commands under `buf alpha` to be deleted at some point. Having `buf alpha protoc` is useful for `buf` development at times as when we get a compilation issue, we can directly compare `protoc` and `buf alpha protoc` results to see if the issue is also present in `protoc`. If this usefulness wanes over time, and/or we have sufficient need to delete the divergent codepaths, we can delete it outright.

An alternative was to split this out into a separate `buf-protoc` or `protoc-buf` binary, that either lives here or in another repository, but this would seem to incur even more maintenance.

Few users are using `buf protoc` itself, and the mitigation is trivial: go back to using mainline `protoc`. `buf protoc` itself requires mainline `protoc` to be installed anyways if using any of the builtin plugins (`--java_out, --cpp_out`, etc).


Patch:
diff --git a/.dockerignore b/.dockerignore
index 59bffcdc9..26e764631 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -7,8 +7,8 @@
 cmd/buf/buf
 cmd/protoc-gen-buf-breaking/protoc-gen-buf-breaking
 cmd/protoc-gen-buf-lint/protoc-gen-buf-lint
-private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver/protoc-gen-insertion-point-receiver
-private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer/protoc-gen-insertion-point-writer
+private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver/protoc-gen-insertion-point-receiver
+private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer/protoc-gen-insertion-point-writer
 private/buf/cmd/buf/workspacetests/other/proto/workspacetest/cache/
 private/bufpkg/bufprotoplugin/cmd/protoc-gen-go-api/protoc-gen-go-api
 private/bufpkg/bufprotoplugin/cmd/protoc-gen-go-apiclient/protoc-gen-go-apiclient
diff --git a/.gitignore b/.gitignore
index 2af697fb1..2784c1806 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,8 +7,8 @@
 /cmd/buf/buf
 /cmd/protoc-gen-buf-breaking/protoc-gen-buf-breaking
 /cmd/protoc-gen-buf-lint/protoc-gen-buf-lint
-/private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver/protoc-gen-insertion-point-receiver
-/private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer/protoc-gen-insertion-point-writer
+/private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver/protoc-gen-insertion-point-receiver
+/private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer/protoc-gen-insertion-point-writer
 /private/buf/cmd/buf/workspacetests/other/proto/workspacetest/cache/
 /private/bufpkg/bufprotoplugin/cmd/protoc-gen-go-api/protoc-gen-go-api
 /private/bufpkg/bufprotoplugin/cmd/protoc-gen-go-apiclient/protoc-gen-go-apiclient
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 43fb919d6..914d62deb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,13 @@
   `--include-imports`, this will result in the [Well-Known Types](https://github.com/bufbuild/wellknowntypes/tree/11ea259bf71c4d386131c1986ffe103cb1edb3d6/v3.19.4/google/protobuf)
   being generated as well. Most language runtimes have the Well-Known Types included as part
   of the core library, making generating the Well-Known Types separately undesirable.
+- Remove `buf protoc`. This was a pre-v1.0 demonstration to show that `buf` compilation
+  produces equivalent results to mainline `protoc`, however `buf` is working on building
+  a better Protobuf future that provides easier mechanics than our former `protoc`-based
+  world. `buf protoc` itself added no benefit over mainline `protoc` beyond being considerably
+  faster and allowing parallel compilation. If `protoc` is required, move back to mainline `protoc`
+  until you can upgrade to `buf`. See [#915](https://github.com/bufbuild/buf/pull/915) for more
+  details.
 
 ## [v1.0.0-rc12] - 2022-02-01
 
diff --git a/make/buf/all.mk b/make/buf/all.mk
index 7f9d3d5ae..9c98c52f8 100644
--- a/make/buf/all.mk
+++ b/make/buf/all.mk
@@ -16,8 +16,8 @@ GO_BINS := $(GO_BINS) \
 	private/pkg/licenseheader/cmd/license-header \
 	private/pkg/spdx/cmd/spdx-go-data
 GO_TEST_BINS := $(GO_TEST_BINS) \
-	private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver \
-	private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer
+	private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver \
+	private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer
 DOCKER_BINS := $(DOCKER_BINS) buf
 FILE_IGNORES := $(FILE_IGNORES) \
 	.build/ \
diff --git a/private/buf/cmd/buf/buf.go b/private/buf/cmd/buf/buf.go
index 865a34296..cc6dacd66 100644
--- a/private/buf/cmd/buf/buf.go
+++ b/private/buf/cmd/buf/buf.go
@@ -19,6 +19,7 @@ import (
 	"time"
 
 	"github.com/bufbuild/buf/private/buf/bufcli"
+	"github.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/protoc"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokencreate"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokendelete"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokenget"
@@ -65,7 +66,6 @@ import (
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modopen"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modprune"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modupdate"
-	"github.com/bufbuild/buf/private/buf/cmd/buf/command/protoc"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/push"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/registry/registrylogin"
 	"github.com/bufbuild/buf/private/buf/cmd/buf/command/registry/registrylogout"
@@ -125,7 +125,6 @@ func NewRootCommand(name string) *appcmd.Command {
 			lint.NewCommand("lint", builder),
 			breaking.NewCommand("breaking", builder),
 			generate.NewCommand("generate", builder),
-			protoc.NewCommand("protoc", builder),
 			lsfiles.NewCommand("ls-files", builder),
 			push.NewCommand("push", builder),
 			{
@@ -284,9 +283,10 @@ func NewRootCommand(name string) *appcmd.Command {
 			},
 			{
 				Use:    "alpha",
-				Short:  "Alpha commands. Unstable and recommended only for experimentation.",
+				Short:  "Alpha commands. Unstable and recommended only for experimentation. These may be deleted.",
 				Hidden: true,
 				SubCommands: []*appcmd.Command{
+					protoc.NewCommand("protoc", builder),
 					{
 						Use:   "registry",
 						Short: "Manage assets on the Buf Schema Registry.",
diff --git a/private/buf/cmd/buf/command/protoc/testdata/1/flags.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/1/flags.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/1/flags.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/1/flags.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/2/flags1.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/2/flags1.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/2/flags1.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/2/flags1.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/2/flags2.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/2/flags2.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/2/flags2.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/2/flags2.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/3/flags1.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/3/flags1.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/3/flags1.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/3/flags1.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/3/flags2.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/3/flags2.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/3/flags2.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/3/flags2.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/insertion/test.proto b/private/buf/cmd/buf/command/alpha/protoc/testdata/insertion/test.proto
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/insertion/test.proto
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/insertion/test.proto
diff --git a/private/buf/cmd/buf/command/protoc/testdata/overlap/a/1.proto b/private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/a/1.proto
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/overlap/a/1.proto
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/a/1.proto
diff --git a/private/buf/cmd/buf/command/protoc/testdata/overlap/a/1.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/a/1.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/overlap/a/1.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/a/1.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/overlap/b/1.txt b/private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/b/1.txt
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/overlap/b/1.txt
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/b/1.txt
diff --git a/private/buf/cmd/buf/command/protoc/testdata/overlap/b/2.proto b/private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/b/2.proto
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/testdata/overlap/b/2.proto
rename to private/buf/cmd/buf/command/alpha/protoc/testdata/overlap/b/2.proto
diff --git a/private/buf/cmd/buf/command/protoc/usage.gen.go b/private/buf/cmd/buf/command/alpha/protoc/usage.gen.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/usage.gen.go
rename to private/buf/cmd/buf/command/alpha/protoc/usage.gen.go
diff --git a/private/buf/cmd/buf/command/generate/generate.go b/private/buf/cmd/buf/command/generate/generate.go
index 2a6bba7da..4c63d272e 100644
--- a/private/buf/cmd/buf/command/generate/generate.go
+++ b/private/buf/cmd/buf/command/generate/generate.go
@@ -172,7 +172,7 @@ as "proto/foo" is contained within "proto".
 
 Plugins are invoked in the order they are specified in the template, but each plugin
 has a per-directory parallel invocation, with results from each invocation combined
-before writing the result. This is equivalent behavior to "buf protoc --by_dir".
+before writing the result.
 
 Insertion points are processed in the order the plugins are specified in the template.
 `,
diff --git a/private/bufpkg/bufmodule/bufmodulebuild/bufmodulebuild.go b/private/bufpkg/bufmodule/bufmodulebuild/bufmodulebuild.go
index 442a45cd9..523f7598b 100644
--- a/private/bufpkg/bufmodule/bufmodulebuild/bufmodulebuild.go
+++ b/private/bufpkg/bufmodule/bufmodulebuild/bufmodulebuild.go
@@ -86,7 +86,7 @@ type ModuleIncludeBuilder interface {
 // NewModuleIncludeBuilder returns a new ModuleIncludeBuilder.
 //
 // TODO: we should parse includeDirPaths for modules as well in theory
-// would be nice to be able to do buf protoc -I path/to/dir -I buf.build/foo/bar/v1
+// would be nice to be able to do buf alpha protoc -I path/to/dir -I buf.build/foo/bar/v1
 func NewModuleIncludeBuilder(
 	logger *zap.Logger,
 	storageosProvider storageos.Provider,


Test patch:
diff --git a/private/buf/cmd/buf/command/protoc/const_unix.go b/private/buf/cmd/buf/command/alpha/protoc/const_unix.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/const_unix.go
rename to private/buf/cmd/buf/command/alpha/protoc/const_unix.go
diff --git a/private/buf/cmd/buf/command/protoc/const_windows.go b/private/buf/cmd/buf/command/alpha/protoc/const_windows.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/const_windows.go
rename to private/buf/cmd/buf/command/alpha/protoc/const_windows.go
diff --git a/private/buf/cmd/buf/command/protoc/errors.go b/private/buf/cmd/buf/command/alpha/protoc/errors.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/errors.go
rename to private/buf/cmd/buf/command/alpha/protoc/errors.go
diff --git a/private/buf/cmd/buf/command/protoc/flags.go b/private/buf/cmd/buf/command/alpha/protoc/flags.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/flags.go
rename to private/buf/cmd/buf/command/alpha/protoc/flags.go
diff --git a/private/buf/cmd/buf/command/protoc/flags_test.go b/private/buf/cmd/buf/command/alpha/protoc/flags_test.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/flags_test.go
rename to private/buf/cmd/buf/command/alpha/protoc/flags_test.go
diff --git a/private/buf/cmd/buf/command/protoc/flags_unix.go b/private/buf/cmd/buf/command/alpha/protoc/flags_unix.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/flags_unix.go
rename to private/buf/cmd/buf/command/alpha/protoc/flags_unix.go
diff --git a/private/buf/cmd/buf/command/protoc/flags_windows.go b/private/buf/cmd/buf/command/alpha/protoc/flags_windows.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/flags_windows.go
rename to private/buf/cmd/buf/command/alpha/protoc/flags_windows.go
diff --git a/private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver/main.go b/private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver/main.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver/main.go
rename to private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver/main.go
diff --git a/private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver/usage.gen.go b/private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver/usage.gen.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver/usage.gen.go
rename to private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver/usage.gen.go
diff --git a/private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer/main.go b/private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer/main.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer/main.go
rename to private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer/main.go
diff --git a/private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer/usage.gen.go b/private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer/usage.gen.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer/usage.gen.go
rename to private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer/usage.gen.go
diff --git a/private/buf/cmd/buf/command/protoc/plugin.go b/private/buf/cmd/buf/command/alpha/protoc/plugin.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/plugin.go
rename to private/buf/cmd/buf/command/alpha/protoc/plugin.go
diff --git a/private/buf/cmd/buf/command/protoc/protoc.go b/private/buf/cmd/buf/command/alpha/protoc/protoc.go
similarity index 100%
rename from private/buf/cmd/buf/command/protoc/protoc.go
rename to private/buf/cmd/buf/command/alpha/protoc/protoc.go
diff --git a/private/buf/cmd/buf/command/protoc/protoc_test.go b/private/buf/cmd/buf/command/alpha/protoc/protoc_test.go
similarity index 99%
rename from private/buf/cmd/buf/command/protoc/protoc_test.go
rename to private/buf/cmd/buf/command/alpha/protoc/protoc_test.go
index fb2d65465..37f958781 100644
--- a/private/buf/cmd/buf/command/protoc/protoc_test.go
+++ b/private/buf/cmd/buf/command/alpha/protoc/protoc_test.go
@@ -48,6 +48,7 @@ var buftestingDirPath = filepath.Join(
 	"..",
 	"..",
 	"..",
+	"..",
 	"private",
 	"bufpkg",
 	"buftesting",
diff --git a/private/bufpkg/bufimage/bufimagebuild/builder_unix_test.go b/private/bufpkg/bufimage/bufimagebuild/builder_unix_test.go
index 16394070c..1f1fb387b 100644
--- a/private/bufpkg/bufimage/bufimagebuild/builder_unix_test.go
+++ b/private/bufpkg/bufimage/bufimagebuild/builder_unix_test.go
@@ -32,7 +32,7 @@ import (
 func TestCompareGoogleapis(t *testing.T) {
 	testingextended.SkipIfShort(t)
 	// Don't run in parallel as it allocates a lot of memory
-	// cannot directly compare with source code info as buf protoc creates additional source
+	// cannot directly compare with source code info as buf alpha protoc creates additional source
 	// code infos that protoc does not
 	image := testBuildGoogleapis(t, false)
 	fileDescriptorSet := bufimage.ImageToFileDescriptorSet(image)
diff --git a/windows/test.bash b/windows/test.bash
index 1f5d70bdb..b97c0fb0b 100644
--- a/windows/test.bash
+++ b/windows/test.bash
@@ -30,6 +30,6 @@ PATH="${DOWNLOAD_CACHE}/protoc/bin:${PATH}"
 go install google.golang.org/protobuf/cmd/protoc-gen-go@${PROTOC_GEN_GO_VERSION}
 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@${PROTOC_GEN_GO_GRPC_VERSION}
 go install ./cmd/buf \
-  ./private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-writer \
-  ./private/buf/cmd/buf/command/protoc/internal/protoc-gen-insertion-point-receiver
+  ./private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-writer \
+  ./private/buf/cmd/buf/command/alpha/protoc/internal/protoc-gen-insertion-point-receiver
 go test ./...