Repository: reviewdog/reviewdog (language: go)

Issue:
Check runs gets added to the wrong check suite
I implemented a linter action for yamllint using reviewdog, see #402.

I still have one problem with it, see https://github.com/aslafy-z/reviewdog-yamllint-action/pull/1. The resulting check gets added into the wrong check suite.

![2019-12-31-194324_798x284_scrot](https://user-images.githubusercontent.com/8191198/71630877-d9dfc180-2c05-11ea-8ed0-6a2387a4f9be.png)
![2019-12-31-194553_429x189_scrot](https://user-images.githubusercontent.com/8191198/71630962-42c73980-2c06-11ea-868c-980c17c15de4.png)

I've been looking around and found https://github.community/t5/GitHub-Actions/Github-Actions-Status-Checks-created-on-incorrect-Check-Suite-Id/m-p/36180#M2466 which refer to the same problem.

Any issues on your side?


Discussion / hints:
Thanks for the report! AFAIU, it's a github issue as you pasted a link.
I'm not aware of any workaround nor issue on reviewdog side.
Hi 👋 
Can't we use the Github checks API to identify the right check suite?
[This action](https://github.com/shyim/junit-report-annotations-action/blob/3d2e5374f2b13e70f6f3209a21adfdbc42c466ae/index.js#L26-L32) accepts a jobName as input and uses it to resolve the check to which the annotations would be reported.
Unfortunately no. It means it re-uses the existing check run for github action job and the check result / status will be conflicted. (tested #797)
@haya14busa Sorry for the dumb question, but can you explain what you expected to see in https://github.com/reviewdog/reviewdog/pull/797/checks compared to what is actually showing there? I'm trying to understand what the root issue is here (and if it has possibly been fixed on GitHub's end since you last posted).
I have the same problem.

@rmacklin The problem is, that if you have multiple workflows like this:
<img width="323" alt="Screenshot 2021-08-05 at 09 28 37@2x" src="https://user-images.githubusercontent.com/104180/128309602-798c819b-aa8a-4627-8fe5-5aa280ee2324.png">
the reported checks by reviewdog are put on the first (?) workflow instead of the workflow that initiated Reviewdog.

So in my case, Reviewdog runs in `Continuous Integration` but it's checks are reported under `PHPStan`. 

This is extremely annoying when you have protected branches with required checks. Those will all say in oragen saying "waiting for required check".

This hasn't been fixed on GitHub's side yet. I opened a support ticket with them 🤷.

Latest case: the flake8 check is generated by the python linting workflow, but it falls under our comment outdated diff workflow.
![image (2)](https://user-images.githubusercontent.com/71290498/181062731-ed39cc07-a312-4fb3-b6b8-bea98aea0e09.png)
![image (1)](https://user-images.githubusercontent.com/71290498/181062739-4a369e7e-ae0e-4caa-a06f-3c35f14b0fe7.png)


For posterity: here's the response I got from support a while ago

> It appears the flake8 test is being created manually using the [checks API](https://docs.github.com/en/rest/checks/runs#create-a-check-run). In general, you can only create 1 [check suite](https://docs.github.com/en/rest/guides/getting-started-with-the-checks-api#about-check-suites) against a particular commit, but GitHub Actions is an exception where you can have multiple check suites for a particular commit -one for every workflow run against that commit.
>
>The problem is that if you manually create a check run, it can't be attached to a specific check suite. The community post below discusses this in more detail here:
>
>https://github.com/orgs/community/discussions/24616#discussioncomment-3244677
>
>The Actions team has received feedback on this and there are discussions on having a way to attach a manually created check to a particular checksuite. I have added your voice to the discussion. You will, however, not be able to do this at the moment, sorry.
Would it be possible to add an environment variable to disable using the checks API and fall back to annotations like it does when running on a fork?
 
https://github.com/reviewdog/reviewdog/blob/98f2b8bb53a0f34b3407480c06ab8eb8506f03d4/cienv/github_actions.go#L107-L117

This would enable inline annotations using this code:

https://github.com/reviewdog/reviewdog/blob/98f2b8bb53a0f34b3407480c06ab8eb8506f03d4/cmd/reviewdog/doghouse.go#L194-L200

This would allow Actions to report failures in the correct group
Is there any news regarding this after 3.5 years? Really driving me crazy…

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2dc554c9e0..01c5c47bbd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### :sparkles: Release Note <!-- optional -->
 
 ### :rocket: Enhancements
-- ...
+- [#1623](https://github.com/reviewdog/reviewdog/pull/1623) Add reporter for GitHub PR annotations `github-pr-annotations`
 
 ### :bug: Fixes
 - ...
diff --git a/README.md b/README.md
index c6d6171374..3a818845f2 100644
--- a/README.md
+++ b/README.md
@@ -492,6 +492,18 @@ $ export REVIEWDOG_INSECURE_SKIP_VERIFY=true # set this as you need to skip veri
 See [GitHub Actions](#github-actions) section too if you can use GitHub
 Actions. You can also use public reviewdog GitHub Actions.
 
+### Reporter: GitHub PR Annotations (-reporter=github-pr-annotations)
+
+github-pr-annotations uses the GitHub Actions annotation format to output errors
+and warnings to `stdout` e.g.
+
+```
+::error line=11,col=41,file=app/index.md::[vale] reported by reviewdog 🐶%0A[demo.Spelling] Did you really mean 'boobarbaz'?%0A%0ARaw Output:%0A{"message": "[demo.Spelling] Did you really mean 'boobarbaz'?", "location": {"path": "app/index.md", "range": {"start": {"line": 11, "column": 41}}}, "severity": "ERROR"}
+```
+
+This reporter requires a valid GitHub API token to generate a diff, but will not
+use the token to report errors.
+
 ### Reporter: GitLab MergeRequest discussions (-reporter=gitlab-mr-discussion)
 
 [![gitlab-mr-discussion sample](https://user-images.githubusercontent.com/3797062/41810718-f91bc540-773d-11e8-8598-fbc09ce9b1c7.png)](https://gitlab.com/reviewdog/reviewdog/-/merge_requests/113#note_83411103)
@@ -929,6 +941,7 @@ so reviewdog will use [Check annotation](https://docs.github.com/en/rest/checks/
 | **`github-check`**           | OK      | OK             | OK                      | OK |
 | **`github-pr-check`**        | OK      | OK             | OK                      | OK |
 | **`github-pr-review`**       | OK      | OK             | OK                      | Partially Supported [1] |
+| **`github-pr-annotations`**   | OK      | OK             | OK                      | OK |
 | **`gitlab-mr-discussion`**   | OK      | OK             | OK                      | Partially Supported [2] |
 | **`gitlab-mr-commit`**       | OK      | Partially Supported [2] | Partially Supported [2] | Partially Supported [2] |
 | **`gerrit-change-review`**   | OK      | OK? [3]        | OK? [3]                 | Partially Supported? [2][3] |
diff --git a/cmd/reviewdog/doghouse.go b/cmd/reviewdog/doghouse.go
index 300c1935d5..3cf7560452 100644
--- a/cmd/reviewdog/doghouse.go
+++ b/cmd/reviewdog/doghouse.go
@@ -24,7 +24,7 @@ import (
 	"github.com/reviewdog/reviewdog/service/serviceutil"
 )
 
-func runDoghouse(ctx context.Context, r io.Reader, w io.Writer, opt *option, isProject bool, forPr bool) error {
+func runDoghouse(ctx context.Context, r io.Reader, w io.Writer, opt *option, isProject bool, forPr bool, useAnnotationReporter bool) error {
 	ghInfo, isPr, err := cienv.GetBuildInfo()
 	if err != nil {
 		return err
@@ -41,11 +41,11 @@ func runDoghouse(ctx context.Context, r io.Reader, w io.Writer, opt *option, isP
 	if err != nil {
 		return err
 	}
-	filteredResultSet, err := postResultSet(ctx, resultSet, ghInfo, cli, opt)
+	filteredResultSet, err := postResultSet(ctx, resultSet, ghInfo, cli, opt, useAnnotationReporter)
 	if err != nil {
 		return err
 	}
-	if foundResultShouldReport := reportResults(w, filteredResultSet); foundResultShouldReport {
+	if foundResultShouldReport := reportResults(w, filteredResultSet, useAnnotationReporter); foundResultShouldReport {
 		return errors.New("found at least one result in diff")
 	}
 	return nil
@@ -112,7 +112,7 @@ func checkResultSet(ctx context.Context, r io.Reader, opt *option, isProject boo
 }
 
 func postResultSet(ctx context.Context, resultSet *reviewdog.ResultMap,
-	ghInfo *cienv.BuildInfo, cli client.DogHouseClientInterface, opt *option) (*reviewdog.FilteredResultMap, error) {
+	ghInfo *cienv.BuildInfo, cli client.DogHouseClientInterface, opt *option, useAnnotationReporter bool) (*reviewdog.FilteredResultMap, error) {
 	var g errgroup.Group
 	wd, _ := os.Getwd()
 	gitRelWd, err := serviceutil.GitRelWorkdir()
@@ -141,7 +141,9 @@ func postResultSet(ctx context.Context, resultSet *reviewdog.ResultMap,
 			if err := result.CheckUnexpectedFailure(); err != nil {
 				return err
 			}
-			res, err := cli.Check(ctx, req)
+
+			res, err := cli.Check(ctx, req, !useAnnotationReporter)
+
 			if err != nil {
 				return fmt.Errorf("post failed for %s: %w", name, err)
 			}
@@ -190,7 +192,7 @@ func checkResultToAnnotation(d *rdf.Diagnostic, wd, gitRelWd string) *doghouse.A
 //
 // It returns true if reviewdog should exit with 1.
 // e.g. At least one annotation result is in diff.
-func reportResults(w io.Writer, filteredResultSet *reviewdog.FilteredResultMap) bool {
+func reportResults(w io.Writer, filteredResultSet *reviewdog.FilteredResultMap, useAnnotationReporter bool) bool {
 	if filteredResultSet.Len() != 0 && cienv.HasReadOnlyPermissionGitHubToken() {
 		fmt.Fprintln(os.Stderr, `reviewdog: This GitHub token doesn't have write permission of Review API [1], 
 so reviewdog will report results via logging command [2] and create annotations similar to
@@ -229,13 +231,16 @@ github-pr-check reporter as a fallback.
 			shouldFail = shouldFail || !cienv.IsInGitHubAction() ||
 				!(results.Level == "warning" || results.Level == "info")
 
+			// If using the annotation reporter, the action should fail
+			shouldFail = shouldFail || !useAnnotationReporter
+
 			if foundNumOverall == githubutils.MaxLoggingAnnotationsPerStep {
 				githubutils.WarnTooManyAnnotationOnce()
 				shouldFail = true
 			}
 
 			foundResultPerName = true
-			if cienv.IsInGitHubAction() {
+			if cienv.IsInGitHubAction() || useAnnotationReporter {
 				githubutils.ReportAsGitHubActionsLog(name, results.Level, result.Diagnostic)
 			} else {
 				// Output original lines.
diff --git a/cmd/reviewdog/main.go b/cmd/reviewdog/main.go
index 832bf657f2..a8651f258e 100644
--- a/cmd/reviewdog/main.go
+++ b/cmd/reviewdog/main.go
@@ -272,9 +272,11 @@ func run(r io.Reader, w io.Writer, opt *option) error {
 	default:
 		return fmt.Errorf("unknown -reporter: %s", opt.reporter)
 	case "github-check":
-		return runDoghouse(ctx, r, w, opt, isProject, false)
+		return runDoghouse(ctx, r, w, opt, isProject, false, false)
 	case "github-pr-check":
-		return runDoghouse(ctx, r, w, opt, isProject, true)
+		return runDoghouse(ctx, r, w, opt, isProject, true, false)
+	case "github-pr-annotations":
+		return runDoghouse(ctx, r, w, opt, isProject, true, true)
 	case "github-pr-review":
 		gs, isPR, err := githubService(ctx, opt)
 		if err != nil {
diff --git a/doghouse/appengine/checker.go b/doghouse/appengine/checker.go
index 84ae29a294..7a18e3bde8 100644
--- a/doghouse/appengine/checker.go
+++ b/doghouse/appengine/checker.go
@@ -59,7 +59,7 @@ func (gc *githubChecker) handleCheck(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	res, err := server.NewChecker(&req, gh).Check(ctx)
+	res, err := server.NewChecker(&req, gh).Check(ctx, true)
 	if err != nil {
 		aelog.Errorf(ctx, "failed to run checker: %v", err)
 		w.WriteHeader(http.StatusBadRequest)
diff --git a/doghouse/client/client.go b/doghouse/client/client.go
index 78c27bead3..55b8d33dd2 100644
--- a/doghouse/client/client.go
+++ b/doghouse/client/client.go
@@ -19,7 +19,7 @@ const baseEndpoint = "https://reviewdog.app"
 
 // DogHouseClientInterface is interface for doghouse client.
 type DogHouseClientInterface interface {
-	Check(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error)
+	Check(ctx context.Context, req *doghouse.CheckRequest, sendRequest bool) (*doghouse.CheckResponse, error)
 }
 
 // DogHouseClient is client for doghouse server.
@@ -48,7 +48,7 @@ func New(client *http.Client) *DogHouseClient {
 }
 
 // Check send check requests to doghouse.
-func (c *DogHouseClient) Check(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
+func (c *DogHouseClient) Check(ctx context.Context, req *doghouse.CheckRequest, sendRequest bool) (*doghouse.CheckResponse, error) {
 	checkURL := c.BaseURL.String() + "/check"
 	b, err := json.Marshal(req)
 	if err != nil {
diff --git a/doghouse/client/github_client.go b/doghouse/client/github_client.go
index 8cac2edf80..55461080d1 100644
--- a/doghouse/client/github_client.go
+++ b/doghouse/client/github_client.go
@@ -15,6 +15,6 @@ type GitHubClient struct {
 	Client *github.Client
 }
 
-func (c *GitHubClient) Check(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
-	return server.NewChecker(req, c.Client).Check(ctx)
+func (c *GitHubClient) Check(ctx context.Context, req *doghouse.CheckRequest, makeRequest bool) (*doghouse.CheckResponse, error) {
+	return server.NewChecker(req, c.Client).Check(ctx, makeRequest)
 }
diff --git a/doghouse/server/doghouse.go b/doghouse/server/doghouse.go
index c47c5d0d9b..8608bf0424 100644
--- a/doghouse/server/doghouse.go
+++ b/doghouse/server/doghouse.go
@@ -40,7 +40,7 @@ func NewChecker(req *doghouse.CheckRequest, gh *github.Client) *Checker {
 	return &Checker{req: req, gh: &checkerGitHubClient{Client: gh}}
 }
 
-func (ch *Checker) Check(ctx context.Context) (*doghouse.CheckResponse, error) {
+func (ch *Checker) Check(ctx context.Context, makeRequest bool) (*doghouse.CheckResponse, error) {
 	var filediffs []*diff.FileDiff
 	if ch.req.PullRequest != 0 {
 		var err error
@@ -59,7 +59,16 @@ func (ch *Checker) Check(ctx context.Context) (*doghouse.CheckResponse, error) {
 		filterMode = filter.ModeNoFilter
 	}
 	filtered := filter.FilterCheck(results, filediffs, 1, "", filterMode)
-	check, err := ch.createCheck(ctx)
+
+	// If we're not making a request, we can just return the filtered results.
+	if !makeRequest {
+		return &doghouse.CheckResponse{CheckedResults: filtered}, nil
+	}
+
+	var check *github.CheckRun
+	var err error
+
+	check, err = ch.createCheck(ctx)
 	if err != nil {
 		// If this error is StatusForbidden (403) here, it means reviewdog is
 		// running on GitHub Actions and has only read permission (because it's


Test patch:
diff --git a/cmd/reviewdog/doghouse_test.go b/cmd/reviewdog/doghouse_test.go
index 286694bab8..49c2dc1859 100644
--- a/cmd/reviewdog/doghouse_test.go
+++ b/cmd/reviewdog/doghouse_test.go
@@ -196,11 +196,11 @@ func TestDiagnosticResultSet_NonProject(t *testing.T) {
 
 type fakeDoghouseServerCli struct {
 	client.DogHouseClientInterface
-	FakeCheck func(context.Context, *doghouse.CheckRequest) (*doghouse.CheckResponse, error)
+	FakeCheck func(context.Context, *doghouse.CheckRequest, bool) (*doghouse.CheckResponse, error)
 }
 
-func (f *fakeDoghouseServerCli) Check(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
-	return f.FakeCheck(ctx, req)
+func (f *fakeDoghouseServerCli) Check(ctx context.Context, req *doghouse.CheckRequest, makeRequest bool) (*doghouse.CheckResponse, error) {
+	return f.FakeCheck(ctx, req, makeRequest)
 }
 
 func TestPostResultSet_withReportURL(t *testing.T) {
@@ -212,7 +212,7 @@ func TestPostResultSet_withReportURL(t *testing.T) {
 	)
 
 	fakeCli := &fakeDoghouseServerCli{}
-	fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
+	fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest, makeRequest bool) (*doghouse.CheckResponse, error) {
 		if req.Owner != owner {
 			t.Errorf("req.Owner = %q, want %q", req.Owner, owner)
 		}
@@ -313,7 +313,7 @@ func TestPostResultSet_withReportURL(t *testing.T) {
 	}
 
 	opt := &option{filterMode: filter.ModeAdded}
-	if _, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt); err != nil {
+	if _, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt, true); err != nil {
 		t.Fatal(err)
 	}
 }
@@ -328,7 +328,7 @@ func TestPostResultSet_withoutReportURL(t *testing.T) {
 
 	wantResults := []*filter.FilteredDiagnostic{{ShouldReport: true}}
 	fakeCli := &fakeDoghouseServerCli{}
-	fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
+	fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest, makeRequest bool) (*doghouse.CheckResponse, error) {
 		return &doghouse.CheckResponse{CheckedResults: wantResults}, nil
 	}
 
@@ -338,7 +338,7 @@ func TestPostResultSet_withoutReportURL(t *testing.T) {
 	ghInfo := &cienv.BuildInfo{Owner: owner, Repo: repo, PullRequest: prNum, SHA: sha}
 
 	opt := &option{filterMode: filter.ModeAdded}
-	resp, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt)
+	resp, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt, true)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -381,12 +381,12 @@ func TestPostResultSet_conclusion(t *testing.T) {
 
 	for _, tt := range tests {
 		tt := tt
-		fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
+		fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest, makeRequest bool) (*doghouse.CheckResponse, error) {
 			return &doghouse.CheckResponse{ReportURL: "xxx", Conclusion: tt.conclusion}, nil
 		}
 		opt := &option{filterMode: filter.ModeAdded, failOnError: tt.failOnError}
 		id := fmt.Sprintf("[conclusion=%s, failOnError=%v]", tt.conclusion, tt.failOnError)
-		_, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt)
+		_, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt, true)
 		if tt.wantErr && err == nil {
 			t.Errorf("[%s] want err, but got nil.", id)
 		} else if !tt.wantErr && err != nil {
@@ -404,7 +404,7 @@ func TestPostResultSet_withEmptyResponse(t *testing.T) {
 	)
 
 	fakeCli := &fakeDoghouseServerCli{}
-	fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest) (*doghouse.CheckResponse, error) {
+	fakeCli.FakeCheck = func(ctx context.Context, req *doghouse.CheckRequest, makeRequest bool) (*doghouse.CheckResponse, error) {
 		return &doghouse.CheckResponse{}, nil
 	}
 
@@ -414,7 +414,7 @@ func TestPostResultSet_withEmptyResponse(t *testing.T) {
 	ghInfo := &cienv.BuildInfo{Owner: owner, Repo: repo, PullRequest: prNum, SHA: sha}
 
 	opt := &option{filterMode: filter.ModeAdded}
-	if _, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt); err == nil {
+	if _, err := postResultSet(context.Background(), &resultSet, ghInfo, fakeCli, opt, true); err == nil {
 		t.Error("got no error but want report missing error")
 	}
 }
@@ -453,7 +453,7 @@ func TestReportResults(t *testing.T) {
 		},
 	})
 	stdout := new(bytes.Buffer)
-	foundResultShouldReport := reportResults(stdout, filteredResultSet)
+	foundResultShouldReport := reportResults(stdout, filteredResultSet, false)
 	if !foundResultShouldReport {
 		t.Errorf("foundResultShouldReport = %v, want true", foundResultShouldReport)
 	}
@@ -486,7 +486,7 @@ func TestReportResults_inGitHubAction(t *testing.T) {
 		},
 	})
 	stdout := new(bytes.Buffer)
-	_ = reportResults(stdout, filteredResultSet)
+	_ = reportResults(stdout, filteredResultSet, false)
 	want := `reviewdog: Reporting results for "name1"
 `
 	if got := stdout.String(); got != want {
@@ -528,7 +528,7 @@ func TestReportResults_noResultsShouldReport(t *testing.T) {
 		},
 	})
 	stdout := new(bytes.Buffer)
-	foundResultShouldReport := reportResults(stdout, filteredResultSet)
+	foundResultShouldReport := reportResults(stdout, filteredResultSet, false)
 	if foundResultShouldReport {
 		t.Errorf("foundResultShouldReport = %v, want false", foundResultShouldReport)
 	}
diff --git a/doghouse/client/client_test.go b/doghouse/client/client_test.go
index a39c8847c3..ab5c497d1f 100644
--- a/doghouse/client/client_test.go
+++ b/doghouse/client/client_test.go
@@ -25,7 +25,7 @@ func TestDogHouseClient_Check(t *testing.T) {
 	cli.BaseURL, _ = url.Parse(ts.URL)
 
 	req := &doghouse.CheckRequest{}
-	resp, err := cli.Check(context.Background(), req)
+	resp, err := cli.Check(context.Background(), req, true)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -47,7 +47,7 @@ func TestDogHouseClient_Check_failure(t *testing.T) {
 	cli.BaseURL, _ = url.Parse(ts.URL)
 
 	req := &doghouse.CheckRequest{}
-	_, err := cli.Check(context.Background(), req)
+	_, err := cli.Check(context.Background(), req, true)
 	if err == nil {
 		t.Error("got no error, but want bad request error")
 	}
diff --git a/doghouse/server/doghouse_test.go b/doghouse/server/doghouse_test.go
index 7f33ce96c2..26f698e356 100644
--- a/doghouse/server/doghouse_test.go
+++ b/doghouse/server/doghouse_test.go
@@ -301,7 +301,7 @@ func TestCheck_OK(t *testing.T) {
 		return &github.CheckRun{HTMLURL: github.String(reportURL)}, nil
 	}
 	checker := &Checker{req: req, gh: cli}
-	res, err := checker.Check(context.Background())
+	res, err := checker.Check(context.Background(), true)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -401,7 +401,7 @@ func testOutsideDiff(t *testing.T, outsideDiff bool, filterMode filter.Mode) {
 		return &github.CheckRun{HTMLURL: github.String(reportURL)}, nil
 	}
 	checker := &Checker{req: req, gh: cli}
-	if _, err := checker.Check(context.Background()); err != nil {
+	if _, err := checker.Check(context.Background(), true); err != nil {
 		t.Fatal(err)
 	}
 }
@@ -480,7 +480,7 @@ func TestCheck_OK_multiple_update_runs(t *testing.T) {
 		return &github.CheckRun{HTMLURL: github.String(reportURL)}, nil
 	}
 	checker := &Checker{req: req, gh: cli}
-	if _, err := checker.Check(context.Background()); err != nil {
+	if _, err := checker.Check(context.Background(), true); err != nil {
 		t.Fatal(err)
 	}
 }
@@ -576,7 +576,7 @@ func TestCheck_OK_nonPullRequests(t *testing.T) {
 		return &github.CheckRun{HTMLURL: github.String(reportURL)}, nil
 	}
 	checker := &Checker{req: req, gh: cli}
-	res, err := checker.Check(context.Background())
+	res, err := checker.Check(context.Background(), true)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -597,7 +597,7 @@ func TestCheck_fail_diff(t *testing.T) {
 	}
 	checker := &Checker{req: req, gh: cli}
 
-	if _, err := checker.Check(context.Background()); err == nil {
+	if _, err := checker.Check(context.Background(), true); err == nil {
 		t.Fatalf("got no error, want some error")
 	} else {
 		t.Log(err)
@@ -616,7 +616,7 @@ func TestCheck_fail_invalid_diff(t *testing.T) {
 	}
 	checker := &Checker{req: req, gh: cli}
 
-	if _, err := checker.Check(context.Background()); err == nil {
+	if _, err := checker.Check(context.Background(), true); err == nil {
 		t.Fatalf("got no error, want some error")
 	} else {
 		t.Log(err)
@@ -634,7 +634,7 @@ func TestCheck_fail_check(t *testing.T) {
 	}
 	checker := &Checker{req: req, gh: cli}
 
-	if _, err := checker.Check(context.Background()); err == nil {
+	if _, err := checker.Check(context.Background(), true); err == nil {
 		t.Fatalf("got no error, want some error")
 	} else {
 		t.Log(err)
@@ -656,7 +656,7 @@ func TestCheck_fail_check_with_403(t *testing.T) {
 	}
 	checker := &Checker{req: req, gh: cli}
 
-	resp, err := checker.Check(context.Background())
+	resp, err := checker.Check(context.Background(), true)
 	if err != nil {
 		t.Fatalf("got unexpected error: %v", err)
 	}
diff --git a/project/run_test.go b/project/run_test.go
index 51ba7e1e80..f498e70f2c 100644
--- a/project/run_test.go
+++ b/project/run_test.go
@@ -5,6 +5,7 @@ import (
 	"context"
 	"errors"
 	"os"
+	"runtime"
 	"strings"
 	"testing"
 
@@ -162,7 +163,12 @@ func TestRun(t *testing.T) {
 		} else {
 			t.Log(err)
 		}
-		want := "sh: 1: not-found: not found\n"
+		var want string
+		if runtime.GOOS == "darwin" {
+			want = "sh: not-found: command not found\n"
+		} else {
+			want = "sh: 1: not-found: not found\n"
+		}
 		if got := buf.String(); got != want {
 			t.Errorf("got stderr %q, want %q", got, want)
 		}