What's happening underneath is, the go build command queries https://example.com/purple/red?go-get=1, ... to get the meta tags.  That means, until I check in the packages and go.mod files to the remote repository and make them accessible, I can't build/test. 

What's the easiest way to start a multi-module repo with inter-dependency? 

hyangah
changed the title from
cmd/build: setting up a multi-module single repo is difficult
to
cmd/go: setting up a multi-module single repo is difficultAug 17, 2018

This comment has been minimized. 

A go.mod is like its own little GOPATH.  There is no implicit reference to other nearby modules.  In particular being in one repo does not mean that they all move in lock step and always refer to the code from the same commit.  That's not what users will get either.  If there is an inter-dependency, you need to add it, with something like

This comment has been minimized. 

edited

This issue is closely related to #26241, #25053 and other issues that involving development of multiple modules concurrently and access to local modules. 

In the above example, one caveat is that the 'replace' rule doesn't take effect until there exist the 'require' entry for the replaced module.  That's why I ended up using 'go mod edit -require' which is not meant for use by human. (# 27060) So, in order to add the local inter-dependency, users need to either

commit the updated red, blue modules upstream so they can be go-gettable.  Then transform purple, or

manually edit go.mod to include red and blue entries both in 'require' and 'replace'. 

This comment has been minimized. 

From my perspective, the only "problem" is the requirement on existence of the submodule (and I don't think we can solve that problem).  For me, this was step 1.  But this was only a "problem" in so far as my CI build broke.  Beyond that commit, all was fine. 

This is effectively the crux of my open questions too.  I'm less worried about the "hand-edit" part, because I think tooling can easily fix that.  It's the commit dance that follows that raises the biggest question.  Again, tooling could help, but because of the mix of VCS systems that people may be using, it's not immediately obvious.  With Gerrit things are certainly much easier...

This comment has been minimized. 

As an experience report, our Go code at Square is organized in a similar fashion.  We are still using godep because we were waiting for the Packaging Wars to settle out. 

We have O(dozens) of apps at $REPO/$APPNAME or $REPO/$TEAM/$APPNAME, as well as components like $REPO/service/logging, $REPO/service/feeds, $REPO/service/rpc, etc. 

It doesn't seem as if hosting multiple modules out of a single repo is clean (version tags are per-repo), but we're considering giving each app a separate go.mod file: it would allow each app to choose what version of third-party libraries it uses. 

We're also considering some kind of linter/auto-go.mod-writing tool to ensure that allgo.mod files include things like replace directives for our protobuf fork.  I was intending to have that tool enter replace $REPO/service/foo => ../service/foo directives for each module imported, or perhaps a global replace $REPO => .. to catch all of them.  It's awkward, but not unworkable: those replace directives would be out-of-place in the checked-in code if this were an open-source repo, but since it's our monorepo, everything would probably work okay with them checked in. 

Anyway, just wanted to log a report.  I expect within Google, Go code is still built using blaze, with custom tooling, so I don't expect monorepo experience help from inside :-)

This comment has been minimized. 

It doesn't seem as if hosting multiple modules out of a single repo is clean (version tags are per-repo)

Do you mean your version tags are per-repo?  Because in general multi-module versions work as a result of the submodule prefix on a tag.  For example the repo that results from the submodules (multi-module repo) guide has the following two tags (i.e. versions):

b/v0.1.1
a/v1.0.0

We're also considering some kind of linter/auto-go.mod-writing tool ...

This would presumably be a wrapper around go mod edit -replace... and friends? 

This comment has been minimized. 

Because in general multi-module versions work as a result of the submodule prefix on a tag. 

Ah, I somehow missed that you could namespace version labels like that.  Simple and clear.  Thanks! 

This would presumably be a wrapper around go mod edit -replace... and friends? 

I imagine so, probably.  The more important reason for linting is that we need to catch an app accidentally not using our protobuf fork.  It could result in not redacting sensitive fields in logs, for example. 

This comment has been minimized. 

edited

We're also considering some kind of linter/auto-go.mod-writing tool to ensure that all go.mod files include things like replace directives for our protobuf fork. 

That seems like a problem you could resolve with a GOPROXY HTTP server, at least: it could act as a straight proxy for most packages (or serve 404s after #26334 is fixed), but serve 403s for any module you want to exclude globally.