Repository: bazelbuild/bazelisk
Pull request #226 (2021-03-22 06:12:40)

Bump github.com/bazelbuild/rules_go from 0.25.1 to 0.27.0
Bumps [github.com/bazelbuild/rules_go](https://github.com/bazelbuild/rules_go) from 0.25.1 to 0.27.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/bazelbuild/rules_go/releases">github.com/bazelbuild/rules_go's releases</a>.</em></p>
<blockquote>
<h2>v0.27.0</h2>
<h2>Updated dependencies</h2>
<p>This release updates <code>golang.org/x/tools</code>, which was mistakenly not updated in v0.26.0. Additionally, the latest version of <code>golang.org/x/tools</code> requires <code>golang.org/x/sys/execabs</code>, so that module is added as a new dependency.</p>
<p>As always, you can use higher versions of rules_go's dependencies by declaring them in WORKSPACE before calling <code>go_rules_dependencies</code>. Lower versions may work but are not supported.</p>
<ul>
<li><code>org_golang_x_tools</code> is updated to <code>v0.1.0</code>.</li>
<li><code>org_golang_x_sys</code> is added at <code>master</code> as of 2021-03-17.</li>
</ul>
<h2><code>WORKSPACE</code> code</h2>
<pre><code>load(&quot;@bazel_tools//tools/build_defs/repo:http.bzl&quot;, &quot;http_archive&quot;)
<p>http_archive(
name = &quot;io_bazel_rules_go&quot;,
sha256 = &quot;69de5c704a05ff37862f7e0f5534d4f479418afc21806c887db544a316f3cb6b&quot;,
urls = [
&quot;https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.27.0/rules_go-v0.27.0.tar.gz&quot;,
&quot;https://github.com/bazelbuild/rules_go/releases/download/v0.27.0/rules_go-v0.27.0.tar.gz&quot;,
],
)</p>
<p>load(&quot;<a href="https://github.com/io"><code>@​io</code></a>_bazel_rules_go//go:deps.bzl&quot;, &quot;go_register_toolchains&quot;, &quot;go_rules_dependencies&quot;)</p>
<p>go_rules_dependencies()</p>
<p>go_register_toolchains(version = &quot;1.16.2&quot;)
</code></pre></p>
<h2>v0.26.0</h2>
<h2>Changes</h2>
<ul>
<li><code>//go:embed</code> directives are supported in Go 1.16. Patterns are matched against both static and generated files, including directory artifacts. Embeddable files must be listed in the new <code>embedsrcs</code> attribute. Gazelle v0.23.0 will generate these attributes automatically.</li>
<li><code>go_binary</code> now provides <code>CcInfo</code> in the <code>c-archive</code> and <code>c-shared</code> link modes. This should let <code>cc_library</code> and other rules reference these targets through <code>deps</code> directly. Thanks <a href="https://github.com/steeve"><code>@​steeve</code></a>.</li>
<li><code>TEST_TMPDIR</code> is no longer set to <code>TMPDIR</code> in tests. Thanks <a href="https://github.com/mikedanese"><code>@​mikedanese</code></a>.</li>
<li>Several small improvements in error and diagnostic output.</li>
</ul>
<h2>Compatibility</h2>
<ul>
<li>Bazel 3.5.0 is now required. The previous minimum version was 3.4.0.</li>
</ul>
<h2>Updated dependencies</h2>
<p>As always, you can use higher versions of rules_go's dependencies by declaring them in <code>WORKSPACE</code> before calling <code>go_rules_dependencies</code>. Lower versions may work but are not supported.</p>
<ul>
<li><code>platforms</code> is updated to <code>0.0.4</code>.</li>
<li><code>rules_cc</code> is updated to <code>master</code> as of 2021-03-05.</li>
<li><code>org_golang_x_tools</code> is updated to <code>master</code> as of 2021-03-05.</li>
<li><code>org_golang_google_genproto</code> is updated to <code>master</code> as of 2021-03-05.</li>
<li><code>go_googleapis</code> is updated to <code>master</code> as of 2021-03-05.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/bazelbuild/rules_go/commit/63ed47c03019cc78800071d96edbe6e190f37284"><code>63ed47c</code></a> Set RULES_GO_VERSION to 0.27.0 (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2851">#2851</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/7ced63b764696db519b0c7f562cd4513e6eab332"><code>7ced63b</code></a> Update org_golang_x_tools and add org_golang_x_sys (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2849">#2849</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/0b2afde488a1eefe6ca468029e94e24b927b0cea"><code>0b2afde</code></a> CI: update to Go 1.16.1 (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2842">#2842</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/64bba9f1da89c5825ab386c9abb5ca439e7f9608"><code>64bba9f</code></a> tests/integration/popular_repos: update all other downstream repos (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2840">#2840</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/083916ce5215338a1016d7d1d6ae46cc1e10da6e"><code>083916c</code></a> Update boilerplate for rules_go v0.26.0 and Gazelle v0.26.0 [skip ci] (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2839">#2839</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/ddc5747aa3a52e7703f48281d82ea06985d95628"><code>ddc5747</code></a> Set RULES_GO_VERSION to 0.26.0 (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2838">#2838</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/7c43bccb748daec205821e7158716a273fc9c6e0"><code>7c43bcc</code></a> Update dependencies for v0.26 (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2837">#2837</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/736f4d6f4c67668ba6340b4e77e0181552392db8"><code>736f4d6</code></a> Split up PIE tests. (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2833">#2833</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/dce580563ee43a048c61f505c120c7234e704029"><code>dce5805</code></a> Add ARM-64 version to go_download_sdk test. (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2832">#2832</a>)</li>
<li><a href="https://github.com/bazelbuild/rules_go/commit/ee1740e04a6918046e3e9f228115e65f3a1e309b"><code>ee1740e</code></a> Modify build constraints test to also work with ARM-64. (<a href="https://github-redirect.dependabot.com/bazelbuild/rules_go/issues/2831">#2831</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/bazelbuild/rules_go/compare/v0.25.1...v0.27.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/bazelbuild/rules_go&package-manager=go_modules&previous-version=0.25.1&new-version=0.27.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>


Patch:
diff --git a/core/core.go b/core/core.go
index 24b2e9a..b910ba4 100644
--- a/core/core.go
+++ b/core/core.go
@@ -441,7 +441,8 @@ type issue struct {
 	Labels []label `json:"labels"`
 }
 
-type issueList struct {
+// IssueList is visible for testing
+type IssueList struct {
 	Items []issue `json:"items"`
 }
 
@@ -464,27 +465,51 @@ func getIncompatibleFlags(bazeliskHome, resolvedBazelVersion string) (map[string
 		return nil, fmt.Errorf("invalid version %v", resolvedBazelVersion)
 	}
 	url := "https://api.github.com/search/issues?per_page=100&q=repo:bazelbuild/bazel+label:migration-" + version
-	issuesJSON, err := httputil.MaybeDownload(bazeliskHome, url, "flags-"+version, "list of flags from GitHub", GetEnvOrConfig("BAZELISK_GITHUB_TOKEN"))
+	
+	var issueList IssueList
+
+	merger := func(chunks [][]byte) ([]byte, error) {
+		for _, chunk := range chunks {
+			current, err := ParseIssues(chunk)
+			if err != nil {
+				return nil, err
+			}
+			issueList.Items = append(issueList.Items, current.Items...)
+		}
+		return json.Marshal(issueList)
+	}
+	
+	issuesJSON, err := httputil.MaybeDownload(bazeliskHome, url, "flags-"+version, "list of flags from GitHub", GetEnvOrConfig("BAZELISK_GITHUB_TOKEN"), merger)
 	if err != nil {
 		return nil, fmt.Errorf("could not get issues from GitHub: %v", err)
 	}
 
-	result, err := ScanIssuesForIncompatibleFlags(issuesJSON)
-	return result, err
+	if len(issueList.Items) == 0 {
+		issueList, err = ParseIssues(issuesJSON)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return ScanIssuesForIncompatibleFlags(issueList), nil
+}
+
+// ParseIssues is visible for testing
+func ParseIssues(data []byte) (IssueList, error) {
+	var result IssueList
+	if err := json.Unmarshal(data, &result); err != nil {
+		return result, fmt.Errorf("could not parse JSON into list of issues: %v", err)
+	}
+	return result, nil
 }
 
 // ScanIssuesForIncompatibleFlags is visible for testing.
 // TODO: move this function and its test into a dedicated package.
-func ScanIssuesForIncompatibleFlags(issuesJSON []byte) (map[string]*FlagDetails, error) {
+func ScanIssuesForIncompatibleFlags(issues IssueList) map[string]*FlagDetails {
 	result := make(map[string]*FlagDetails)
-	var issueList issueList
-	if err := json.Unmarshal(issuesJSON, &issueList); err != nil {
-		return nil, fmt.Errorf("could not parse JSON into list of issues: %v", err)
-	}
-
 	re := regexp.MustCompile(`^incompatible_\w+`)
 	sRe := regexp.MustCompile(`^//.*[^/]:incompatible_\w+`)
-	for _, issue := range issueList.Items {
+	for _, issue := range issues.Items {
 		flag := re.FindString(issue.Title)
 		if len(flag) <= 0 {
 			flag = sRe.FindString(issue.Title)
@@ -499,7 +524,7 @@ func ScanIssuesForIncompatibleFlags(issuesJSON []byte) (map[string]*FlagDetails,
 		}
 	}
 
-	return result, nil
+	return result
 }
 
 func getBreakingRelease(labels []label) string {
diff --git a/go.mod b/go.mod
index 9e49ff6..29e4cf5 100644
--- a/go.mod
+++ b/go.mod
@@ -3,7 +3,7 @@ module github.com/bazelbuild/bazelisk
 go 1.15
 
 require (
-	github.com/bazelbuild/rules_go v0.25.1
+	github.com/bazelbuild/rules_go v0.27.0
 	github.com/hashicorp/go-version v1.3.0
 	github.com/mitchellh/go-homedir v1.1.0
 )
diff --git a/go.sum b/go.sum
index 1fc7681..206034c 100644
--- a/go.sum
+++ b/go.sum
@@ -1,5 +1,5 @@
-github.com/bazelbuild/rules_go v0.25.1 h1:kFMT+OgT62SY3tBTRYFJBTphOHHoCfyn5wbRFNauKUQ=
-github.com/bazelbuild/rules_go v0.25.1/go.mod h1:MC23Dc/wkXEyk3Wpq6lCqz0ZAYOZDw2DR5y3N1q2i7M=
+github.com/bazelbuild/rules_go v0.27.0 h1:KViqR7qKXwz+LrNdIauCDU21kneCk+4DnYjpvlJwH50=
+github.com/bazelbuild/rules_go v0.27.0/go.mod h1:MC23Dc/wkXEyk3Wpq6lCqz0ZAYOZDw2DR5y3N1q2i7M=
 github.com/hashicorp/go-version v1.3.0 h1:McDWVJIU/y+u1BRV06dPaLfLCaT7fUTJLp5r04x7iNw=
 github.com/hashicorp/go-version v1.3.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
 github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
diff --git a/httputil/httputil.go b/httputil/httputil.go
index 5a2b09d..d177f45 100644
--- a/httputil/httputil.go
+++ b/httputil/httputil.go
@@ -9,24 +9,26 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
+	"regexp"
 	"time"
 )
 
 var (
 	// DefaultTransport specifies the http.RoundTripper that is used for any network traffic, and may be replaced with a dummy implementation for unit testing.
 	DefaultTransport = http.DefaultTransport
+	linkPattern = regexp.MustCompile(`<(.*?)>; rel="(\w+)"`)
 )
 
 func getClient() *http.Client {
 	return &http.Client{Transport: DefaultTransport}
 }
 
-// ReadRemoteFile returns the contents of the given file, using the supplied Authorization token, if set.
-func ReadRemoteFile(url string, token string) ([]byte, error) {
+// ReadRemoteFile returns the contents of the given file, using the supplied Authorization token, if set. It also returns the HTTP headers.
+func ReadRemoteFile(url string, token string) ([]byte, http.Header, error) {
 	client := getClient()
 	req, err := http.NewRequest("GET", url, nil)
 	if err != nil {
-		return nil, fmt.Errorf("could not create request: %v", err)
+		return nil, nil, fmt.Errorf("could not create request: %v", err)
 	}
 
 	if token != "" {
@@ -35,19 +37,19 @@ func ReadRemoteFile(url string, token string) ([]byte, error) {
 
 	res, err := client.Do(req)
 	if err != nil {
-		return nil, fmt.Errorf("could not fetch %s: %v", url, err)
+		return nil, nil, fmt.Errorf("could not fetch %s: %v", url, err)
 	}
 	defer res.Body.Close()
 
 	if res.StatusCode != 200 {
-		return nil, fmt.Errorf("unexpected status code while reading %s: %v", url, res.StatusCode)
+		return nil, res.Header, fmt.Errorf("unexpected status code while reading %s: %v", url, res.StatusCode)
 	}
 
 	body, err := ioutil.ReadAll(res.Body)
 	if err != nil {
-		return nil, fmt.Errorf("failed to read content at %s: %v", url, err)
+		return nil, res.Header, fmt.Errorf("failed to read content at %s: %v", url, err)
 	}
-	return body, nil
+	return body, res.Header, nil
 }
 
 // DownloadBinary downloads a file from the given URL into the specified location, marks it executable and returns its full path.
@@ -101,12 +103,13 @@ func DownloadBinary(originURL, destDir, destFile string) (string, error) {
 	return destinationPath, nil
 }
 
+type ContentMerger func([][]byte) ([]byte, error)
+
 // MaybeDownload downloads a file from the given url and caches the result under bazeliskHome.
 // It skips the download if the file already exists and is not outdated.
 // Parameter ´description´ is only used to provide better error messages.
-func MaybeDownload(bazeliskHome, url, filename, description, token string) ([]byte, error) {
+func MaybeDownload(bazeliskHome, url, filename, description, token string, merger ContentMerger) ([]byte, error) {
 	cachePath := filepath.Join(bazeliskHome, filename)
-
 	if cacheStat, err := os.Stat(cachePath); err == nil {
 		if time.Since(cacheStat.ModTime()).Hours() < 1 {
 			res, err := ioutil.ReadFile(cachePath)
@@ -117,16 +120,40 @@ func MaybeDownload(bazeliskHome, url, filename, description, token string) ([]by
 		}
 	}
 
-	// We could also use go-github here, but I can't get it to build with Bazel's rules_go and it pulls in a lot of dependencies.
-	body, err := ReadRemoteFile(url, token)
+	contents := make([][]byte, 0)
+	nextUrl := url
+	for nextUrl != "" {
+		// We could also use go-github here, but I can't get it to build with Bazel's rules_go and it pulls in a lot of dependencies.
+		body, headers, err := ReadRemoteFile(nextUrl, token)
+		if err != nil {
+			return nil, fmt.Errorf("could not download %s: %v", description, err)
+		}
+		contents = append(contents, body)
+		nextUrl = getNextUrl(headers)
+	}
+
+	merged, err := merger(contents)
 	if err != nil {
-		return nil, fmt.Errorf("could not download %s: %v", description, err)
+		return nil, fmt.Errorf("failed to merge %d chunks from %s: %v", len(contents), url, err)
 	}
 
-	err = ioutil.WriteFile(cachePath, body, 0666)
+	err = ioutil.WriteFile(cachePath, merged, 0666)
 	if err != nil {
 		return nil, fmt.Errorf("could not create %s: %v", cachePath, err)
 	}
 
-	return body, nil
+	return merged, nil
+}
+
+func getNextUrl(headers http.Header) string {
+	links := headers["Link"]
+	if len(links) != 1 {
+		return ""
+	}
+	for _, m := range linkPattern.FindAllStringSubmatch(links[0], -1) {
+		if m[2] == "next" {
+			return m[1]
+		}
+	}	
+	return ""
 }
diff --git a/repositories/gcs.go b/repositories/gcs.go
index 6f74b36..82e33dd 100644
--- a/repositories/gcs.go
+++ b/repositories/gcs.go
@@ -76,7 +76,7 @@ func listDirectoriesInReleaseBucket(prefix string) ([]string, bool, error) {
 		if nextPageToken != "" {
 			url = fmt.Sprintf("%s&pageToken=%s", baseURL, nextPageToken)
 		}
-		content, err := httputil.ReadRemoteFile(url, "")
+		content, _, err := httputil.ReadRemoteFile(url, "")
 		if err != nil {
 			return nil, false, fmt.Errorf("could not list GCS objects at %s: %v", url, err)
 		}
@@ -226,7 +226,7 @@ func (gcs *GCSRepo) DownloadCandidate(version, destDir, destFile string) (string
 // it's https://buildkite.com/bazel/bazel-bazel
 func (gcs *GCSRepo) GetLastGreenCommit(bazeliskHome string, downstreamGreen bool) (string, error) {
 	pathSuffix := lastGreenCommitPathSuffixes[downstreamGreen]
-	content, err := httputil.ReadRemoteFile(lastGreenBaseURL+pathSuffix, "")
+	content, _, err := httputil.ReadRemoteFile(lastGreenBaseURL+pathSuffix, "")
 	if err != nil {
 		return "", fmt.Errorf("could not determine last green commit: %v", err)
 	}
diff --git a/repositories/github.go b/repositories/github.go
index 4fab351..e3134cc 100644
--- a/repositories/github.go
+++ b/repositories/github.go
@@ -31,15 +31,38 @@ func (gh *GitHubRepo) GetVersions(bazeliskHome, bazelFork string) ([]string, err
 }
 
 func (gh *GitHubRepo) getFilteredVersions(bazeliskHome, bazelFork string, wantPrerelease bool) ([]string, error) {
+	parse := func(data []byte) ([]gitHubRelease, error) {
+		var releases []gitHubRelease
+		if err := json.Unmarshal(data, &releases); err != nil {
+			return nil, fmt.Errorf("could not parse JSON into list of releases: %v", err)
+		}
+		return releases, nil
+	}
+
+	var releases []gitHubRelease
+
+	merger := func(chunks [][]byte) ([]byte, error) {
+		for _, chunk := range chunks {
+			current, err := parse(chunk)
+			if err != nil {
+				return nil, err
+			}
+			releases = append(releases, current...)
+		}
+		return json.Marshal(releases)
+	}
+
 	url := fmt.Sprintf("https://api.github.com/repos/%s/bazel/releases", bazelFork)
-	releasesJSON, err := httputil.MaybeDownload(bazeliskHome, url, bazelFork+"-releases.json", "list of Bazel releases from github.com/"+bazelFork, gh.token)
+	releasesJSON, err := httputil.MaybeDownload(bazeliskHome, url, bazelFork+"-releases.json", "list of Bazel releases from github.com/"+bazelFork, gh.token, merger)
 	if err != nil {
 		return []string{}, fmt.Errorf("unable to dermine '%s' releases: %v", bazelFork, err)
 	}
 
-	var releases []gitHubRelease
-	if err := json.Unmarshal(releasesJSON, &releases); err != nil {
-		return []string{}, fmt.Errorf("could not parse JSON into list of releases: %v", err)
+	if len(releases) == 0 {
+		releases, err = parse(releasesJSON)
+		if err != nil {
+			return nil, err
+		}
 	}
 
 	var tags []string


Test patch:
diff --git a/bazelisk_test.go b/bazelisk_test.go
index cb612b6..63448f3 100644
--- a/bazelisk_test.go
+++ b/bazelisk_test.go
@@ -14,10 +14,12 @@ func TestScanIssuesForIncompatibleFlags(t *testing.T) {
 	if err != nil {
 		t.Errorf("Can not load sample github issues")
 	}
-	flags, err := core.ScanIssuesForIncompatibleFlags(samplesJSON)
-	if err != nil || flags == nil {
-		t.Errorf("Could not parse sample issues")
+	issues, err := core.ParseIssues(samplesJSON)
+	if err != nil {
+		t.Errorf("Can not parse sample github issues: %v", err)
 	}
+
+	flags := core.ScanIssuesForIncompatibleFlags(issues)
 	expectedFlagnames := []string{
 		"--//some/path:incompatible_user_defined_flag",
 		"--incompatible_always_check_depset_elements",