Repository: pantsbuild/pants

Issue:
Pants Go support does not handle internal packages.
For example, Go code with a dependency on `cloud.google.com/go/pubsub` needs a BUILD file like the following under `3rdaprty/go` (assuming that's the source root where 3rdparty go BUILD files are housed) @ `cloud.google.com/go/pubsub/BUILD`
```
go_remote_libraries(
  rev='v0.77.0',
  packages=[
    'pubsub',
  ]
)
```

When Pants attempts to resolve that remote library it recurses on its imports - which is good - but it does not recognize one of its imports, `cloud.google.com/go/internal/pubsub` as coming from the same remote root at `cloud.google.com/go` - which is bad. This leads to an attempt to fetch the `cloud.google.com/go/internal/pubsub` which nets:
```
19:20:49 [ERROR] No <meta name="go-import"> tag found at cloud.google.com/go/internal/pubsub
```
Which is correct. Pants should attempt no fetch here.


Discussion:
The relevant docs are here: https://golang.org/cmd/go/#hdr-Remote_import_paths

The key bit snipped:

>  If the import path is not a known code hosting site and also lacks a version control qualifier, the go tool attempts to fetch the import over https/http and looks for a <meta> tag in the document's HTML <head>.
>
> The meta tag has the form:
>
> `<meta name="go-import" content="import-prefix vcs repo-root">`
>
> The import-prefix is the import path corresponding to the repository root. It must be a prefix or an exact match of the package being fetched with "go get". If it's not an exact match, another http request is made at the prefix to verify the <meta> tags match. 

The last three sentences assure us that, for example, `cloud.google.com/go/internal/pubsub` must come from the same repo or archive as `cloud.google.com/go/pubsub`. In short, the meta tag `import-prefix` (i.e. the go remote library remote root) can be used to short circuit fetches for any contained packages whether they be special `internal`, `vendor` or normal.
And what turned out to be tangentially related docs in the path here for the special directories:
+ internal: https://golang.org/cmd/go/#hdr-Internal_Directories
+ vendor: https://golang.org/cmd/go/#hdr-Vendor_Directories

Resolving patch:
diff --git a/contrib/go/src/python/pants/contrib/go/subsystems/fetcher.py b/contrib/go/src/python/pants/contrib/go/subsystems/fetcher.py
--- a/contrib/go/src/python/pants/contrib/go/subsystems/fetcher.py
+++ b/contrib/go/src/python/pants/contrib/go/subsystems/fetcher.py
@@ -62,7 +62,7 @@ class CloningFetcher(Fetcher):
     https://golang.org/cmd/go/#hdr-Remote_import_paths. In particular, it inspects go-import
     meta tags.
 
-    Not that currently we require meta tags, and don't support the explicit form:
+    Note that currently we require meta tags, and don't support the explicit form:
     import "example.org/repo.git/foo/bar", as it looks like it's not used much in practice.
     """
 
@@ -76,8 +76,34 @@ def root(self):
         imported_repo = self._meta_tag_reader.get_imported_repo(self.import_path)
         if imported_repo:
             return imported_repo.import_prefix
+
+        # The import path is for an internal package which is described here:
+        #   https://golang.org/cmd/go/#hdr-Internal_Directories.
+        #
+        # Since an internal directory is only importable in the tree rooted at its parent, we
+        # assume its parent must have a public meta tag page. Failing that, we crawl all the way
+        # back up the ancestry.
+        import_path_components = self.import_path.split("/")
+        try:
+            internal_index = import_path_components.index("internal")
+        except ValueError:
+            internal_import_public_parent_candidates = []
         else:
-            raise FetchError(f'No <meta name="go-import"> tag found at {self.import_path}')
+            internal_import_public_parent_candidates = import_path_components[:internal_index]
+
+        while internal_import_public_parent_candidates:
+            parent_import_path = "/".join(internal_import_public_parent_candidates)
+            imported_repo = self._meta_tag_reader.get_imported_repo(parent_import_path)
+            if imported_repo:
+                logger.info(
+                    "Found public root for private internal import {} in parent {}.".format(
+                        self.import_path, parent_import_path
+                    )
+                )
+                return imported_repo.import_prefix
+            internal_import_public_parent_candidates.pop()
+
+        raise FetchError(f'No <meta name="go-import"> tag found at {self.import_path}')
 
     def fetch(self, dest, rev=None):
         imported_repo = self._meta_tag_reader.get_imported_repo(self.import_path)


Test patch:
diff --git a/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_fetchers.py b/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_fetchers.py
--- a/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_fetchers.py
+++ b/contrib/go/tests/python/pants_test/contrib/go/subsystems/test_fetchers.py
@@ -4,6 +4,7 @@
 from pants.testutil.subsystem.util import global_subsystem_instance
 from pants.testutil.test_base import TestBase
 
+from pants.contrib.go.subsystems.fetch_error import FetchError
 from pants.contrib.go.subsystems.fetcher_factory import FetcherFactory
 
 
@@ -39,3 +40,19 @@ def test_default_golang(self):
 
     def test_default_gopkg(self):
         self.check_default("gopkg.in/check.v1", expected_root="gopkg.in/check.v1")
+
+    def test_default_cloud_google_com(self):
+        self.check_default("cloud.google.com/go/pubsub", expected_root="cloud.google.com/go")
+
+    def test_default_cloud_google_com_issues_11579(self):
+        # This internal package has no meta tag page / is not publicly addressable. Test that
+        # we fall back to searching for the internal package's root in parents.
+        self.check_default(
+            "cloud.google.com/go/internal/pubsub", expected_root="cloud.google.com/go"
+        )
+
+        with self.assertRaises(FetchError):
+            self.fetcher("internal").root()
+
+        with self.assertRaises(FetchError):
+            self.fetcher("host/internal").root()