Repository: gotestyourself/gotestsum
Pull request #258 (2022-06-20 01:20:45)

Add tool ci-matrix
This PR adds a new subcommand `gotestsum tool ci-matrix` for use with github actions.  The subcommand:
* reads `test2json` files saved in a github actions cache from previous test runs
* uses the `test2json` data to calculate the 85th percentile runtime of each package
* uses the package runtime to bucket all the packages into `n` buckets

A github actions workflow can then use those buckets in a [matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs), so that packages are split between the buckets based on their runtime. This should result in optimal splitting of packages to minimize overall CI runtime.

Using the subcommand requires integration with github actions, so I'll need to document exact how to wire it all together.  Once it's working well, maybe it would make sense to publish an action to handle parts of it.

AaronFriel: Hey @dnephin, I'd like to look at using this to improve CI on pulumi/pulumi. Any way I can help review this, validate it?
dnephin: They @AaronFriel , that would be great! The main project I'm working on these days has a test suites that is just fast enough that it doesn't benefit from this much (at least not yet).  It would be great to try it out on a larger project with more tests.

My first trial run is here: https://github.com/dnephin/infra/commit/48d0c57f08203b6fc453c13868dfc267d4912747, but that was a while ago.  If you want to try it out based on that config, make sure to update the `gotestsum` git sha to 7f0dfd73d2c769fdc8c6c5fc3b62c6eec94ebb04.

I'll work on getting this running on a schedule on my fork to see how test runtime improves as the cache collects more data from previous runs. Once that's done I'll share a link to the updated config in this PR.

If you have any questions about how this works, or how to use it, I'd be happy to answer them. 

I'm ready to merge this as soon as I've seen it run well in a real CI workflow. If you have any suggestions for the code, or the feature, please do share!
dnephin: This diff (https://github.com/dnephin/infra/compare/5c7b7459267b91daea2b3e2ecd868522795f3e50..try-test-matrix-2) seems to be working well.

It adds 3 things:
* a `test-matrix` job that uses this `gotestsum tool ci-matrix` command to create the JSON matrix using previous test timing. The matrix is a job output.
* the `test` job is updated to use that matrix and to output timing reports as artifacts.
* the `test-collect-reports` job runs after `test` to download the artifacts from each of the parallel jobs, and save them into a cache, so that the next run of `test-matrix` can use the test reports.
AaronFriel: @dnephin I definitely have a larger, and more expensive, matrix of tests we want to run this on ([recent run](https://github.com/pulumi/pulumi/actions/runs/2982884175)). We've been relying on breaking out the tests by hand, but maintaining that by hand is a bit frustrating and doesn't offer opportunities for splitting tests into more groups.

How would this branch handle adding a new package in a PR, that wasn't in the previous test run?

I think in terms of using it, we had the same thought. Using the actions cache to restore before and after. We'll bake in a default profile into a JSON file in the repo from a prior run or a local run to ensure we have some data.

For pulumi/pulumi to use this, could you push a tag like `ci-matrix-0.1` for us use for security and reliability? In my experience branches are mutable refs & when github prunes commits (because a branch was deleted, for example) `go install ...@commitish` can fail.
AaronFriel: Oh - another thing, is there a more compact representation of the json reports, a simplify/prune/distill the information as a preprocessing step? We produce roughly 3.5GiB of JSON output, which is a little over a third of the cache capacity of a GitHub repo. We'd obliterate all our other caches in 3 runs.

If we could run a `jq` pass - or even better, use the result of `ci-matrix` as the cached output between runs?

```
ls ../test-reports/ -lah
total 3.5G
drwxr-xr-x  2 friel friel 4.0K Sep  3 21:09 .
drwxr-xr-x 19 friel friel 4.0K Sep  3 21:09 ..
-rw-r--r--  1 friel friel  28K Sep  3 19:47 438ea576-9267-47fe-a023-b5fc4814deaf.json
-rw-r--r--  1 friel friel 1.2G Sep  3 19:56 5eabe5e9-db9d-48bd-bd13-7913c4eeb3e4.json
-rw-r--r--  1 friel friel  28K Sep  3 19:05 66073323-df78-4be8-b5df-224d5a148e14.json
-rw-r--r--  1 friel friel 3.0M Sep  3 19:25 874f76fe-2f2a-40bc-94b1-3ac53f7deb59.json
-rw-r--r--  1 friel friel 2.9M Sep  3 19:50 9685a1e1-e7da-42d2-97f3-62d59606bfcb.json
-rw-r--r--  1 friel friel  27K Sep  3 19:45 992d418b-ae8e-4dbe-ae08-e213c2628011.json
-rw-r--r--  1 friel friel  18K Sep  3 19:49 9b3ee554-2c7d-4931-82a0-744573de956d.json
-rw-r--r--  1 friel friel  18K Sep  3 19:08 a4b9a0ab-a257-4572-8a87-f7a9938f40a6.json
-rw-r--r--  1 friel friel  18K Sep  3 19:24 c78f5ecd-a86a-447d-ad23-481a1e43ae13.json
-rw-r--r--  1 friel friel 1.2G Sep  3 19:35 d7387e6b-c9ad-450b-94cd-0aa30e875b55.json
-rw-r--r--  1 friel friel  28K Sep  3 19:22 e7e22942-f20c-46ee-ab3a-4c1f9aaeb8ed.json
-rw-r--r--  1 friel friel 2.9M Sep  3 19:09 ed80e727-74c7-40e5-80e3-51da252b4f1a.json
-rw-r--r--  1 friel friel 1.2G Sep  3 19:20 f98ed980-ff60-47c7-97bd-c5b0063d13f2.json
```
dnephin: > How would this branch handle adding a new package in a PR, that wasn't in the previous test run?

Excellent question!  To ensure that all tests are run, the full list of tests is always pulled from stdin. It should always be run this way:

```sh
go list ./... | gotestsum tool ci-matrix --debug --dir=
```

If a package has no timing data associated with it, it will be assigned to the bucket with the smallest about of estimated runtime.

> We'll bake in a default profile into a JSON file in the repo from a prior run or a local run to ensure we have some data.

Probably a good idea. In the case where there is no timing data, `gotestsum tool ci-matrix` will round-robin the packages into buckets. So you still end up with some improvements over running them all sequentially, but it won't be optimal.

> For pulumi/pulumi to use this, could you push a tag like ci-matrix-0.1 for us use for security and reliability?

Yes, no problem. I'll tag a pre-release with these changes by early next week. I'd like to make a few small improvements before tagging it.
dnephin: > is there a more compact representation of the json reports, a simplify/prune/distill the information as a preprocessing step? 

Oh ya, I was expected this to be a problem for larger projects. Awesome that you found a way to reduce the size of the logs. I've worked on projects that output hundreds of lines of logs per test, which produced similarly large output.

I haven't solved that problem yet, but I have a couple of ideas for how it might be solved.

1. One option would be to add a new command line flag to the main `gotestsum` command to tell it to only output events that are not `Action=Output`. Something like `--jsonfile-omit-output` (boolean flag that would have to be used in addition to `--jsonfile`), or `--jsonfile-timing=<filename>` (a separate flag that writes only the important timing events, and none of the other events).
2. The second option would be a new filter subcommand `gotestsum tool filter` , that would do basically the same thing. Accept a full log of events, and filter it based on the flags passed to that command.

`--jsonfile-timing=<filename>` seems like it might be the easiest to use.
dnephin: I had a quick look at your CI workflow. If you are looking to split a single package by test runtime, that is not supported yet. This command splits by package which is a bit easier.

I think it would be possible to add support for splitting a single package by test timing, but I think it'd be better to do that as a second PR after this one merges.
dnephin: This is the [updated example](https://github.com/dnephin/infra/compare/5c7b7459267b91daea2b3e2ecd868522795f3e50..try-test-matrix) with the latest changes in this branch. The command now outputs a full matrix json, which makes it easier to use in the dependent test job.
dnephin: I've started that work in #277, just needs a bit more testing I think
AaronFriel: I'm already having tremendous success on the previous revision, example here: https://github.com/pulumi/pulumi/actions/runs/2996070671

Looking forward to this stabilizing.

> `--jsonfile-timing=<filename>` seems like it might be the easiest to use.

Agreed. This would also help in case we inadvertently log secrets in a test. GitHub Actions will redact, but these test files would be raw output, I believe?

Patch:
diff --git a/cmd/tool/matrix/matrix.go b/cmd/tool/matrix/matrix.go
new file mode 100644
index 0000000..b5db797
--- /dev/null
+++ b/cmd/tool/matrix/matrix.go
@@ -0,0 +1,274 @@
+package matrix
+
+import (
+	"bufio"
+	"encoding/json"
+	"fmt"
+	"io"
+	"math"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/dnephin/pflag"
+	"gotest.tools/gotestsum/internal/log"
+	"gotest.tools/gotestsum/testjson"
+)
+
+func Run(name string, args []string) error {
+	flags, opts := setupFlags(name)
+	switch err := flags.Parse(args); {
+	case err == pflag.ErrHelp:
+		return nil
+	case err != nil:
+		usage(os.Stderr, name, flags)
+		return err
+	}
+	opts.stdin = os.Stdin
+	opts.stdout = os.Stdout
+	return run(*opts)
+}
+
+type options struct {
+	numPartitions      uint
+	timingFilesPattern string
+	debug              bool
+
+	// shims for testing
+	stdin  io.Reader
+	stdout io.Writer
+}
+
+func setupFlags(name string) (*pflag.FlagSet, *options) {
+	opts := &options{}
+	flags := pflag.NewFlagSet(name, pflag.ContinueOnError)
+	flags.SetInterspersed(false)
+	flags.Usage = func() {
+		usage(os.Stdout, name, flags)
+	}
+	flags.UintVar(&opts.numPartitions, "partitions", 0,
+		"number of parallel partitions to create in the test matrix")
+	flags.StringVar(&opts.timingFilesPattern, "timing-files", "",
+		"glob pattern to match files that contain test2json events, ex: ./logs/*.log")
+	flags.BoolVar(&opts.debug, "debug", false,
+		"enable debug logging")
+	return flags, opts
+}
+
+func usage(out io.Writer, name string, flags *pflag.FlagSet) {
+	fmt.Fprintf(out, `Usage:
+    %[1]s [flags]
+
+Read a list of packages from stdin and output a GitHub Actions matrix strategy
+that splits the packages by previous run times to minimize overall CI runtime.
+
+    echo -n "matrix=" >> $GITHUB_OUTPUT
+    go list ./... | %[1]s --timing-files ./*.log --partitions 4 >> $GITHUB_OUTPUT
+
+The output of the command is a JSON object that can be used as the matrix
+strategy for a test job.
+
+
+Flags:
+`, name)
+	flags.SetOutput(out)
+	flags.PrintDefaults()
+}
+
+func run(opts options) error {
+	log.SetLevel(log.InfoLevel)
+	if opts.debug {
+		log.SetLevel(log.DebugLevel)
+	}
+	if opts.numPartitions < 2 {
+		return fmt.Errorf("--partitions must be atleast 2")
+	}
+	if opts.timingFilesPattern == "" {
+		return fmt.Errorf("--timing-files is required")
+	}
+
+	pkgs, err := readPackages(opts.stdin)
+	if err != nil {
+		return fmt.Errorf("failed to read packages from stdin: %v", err)
+	}
+
+	files, err := readTimingReports(opts)
+	if err != nil {
+		return fmt.Errorf("failed to read or delete timing files: %v", err)
+	}
+	defer closeFiles(files)
+
+	pkgTiming, err := packageTiming(files)
+	if err != nil {
+		return err
+	}
+
+	buckets := bucketPackages(packagePercentile(pkgTiming), pkgs, opts.numPartitions)
+	return writeMatrix(opts.stdout, buckets)
+}
+
+func readPackages(stdin io.Reader) ([]string, error) {
+	var packages []string
+	scan := bufio.NewScanner(stdin)
+	for scan.Scan() {
+		packages = append(packages, scan.Text())
+	}
+	return packages, scan.Err()
+}
+
+func readTimingReports(opts options) ([]*os.File, error) {
+	fileNames, err := filepath.Glob(opts.timingFilesPattern)
+	if err != nil {
+		return nil, err
+	}
+
+	files := make([]*os.File, 0, len(fileNames))
+	for _, fileName := range fileNames {
+		fh, err := os.Open(fileName)
+		if err != nil {
+			return nil, err
+		}
+		files = append(files, fh)
+	}
+
+	log.Infof("Found %v timing files in %v", len(files), opts.timingFilesPattern)
+	return files, nil
+}
+
+func parseEvent(reader io.Reader) (testjson.TestEvent, error) {
+	event := testjson.TestEvent{}
+	err := json.NewDecoder(reader).Decode(&event)
+	return event, err
+}
+
+func packageTiming(files []*os.File) (map[string][]time.Duration, error) {
+	timing := make(map[string][]time.Duration)
+	for _, fh := range files {
+		exec, err := testjson.ScanTestOutput(testjson.ScanConfig{Stdout: fh})
+		if err != nil {
+			return nil, fmt.Errorf("failed to read events from %v: %v", fh.Name(), err)
+		}
+
+		for _, pkg := range exec.Packages() {
+			timing[pkg] = append(timing[pkg], exec.Package(pkg).Elapsed())
+		}
+	}
+	return timing, nil
+}
+
+func packagePercentile(timing map[string][]time.Duration) map[string]time.Duration {
+	result := make(map[string]time.Duration)
+	for pkg, times := range timing {
+		lenTimes := len(times)
+		if lenTimes == 0 {
+			result[pkg] = 0
+			continue
+		}
+
+		sort.Slice(times, func(i, j int) bool {
+			return times[i] < times[j]
+		})
+
+		r := int(math.Ceil(0.85 * float64(lenTimes)))
+		if r == 0 {
+			result[pkg] = times[0]
+			continue
+		}
+		result[pkg] = times[r-1]
+	}
+	return result
+}
+
+func closeFiles(files []*os.File) {
+	for _, fh := range files {
+		_ = fh.Close()
+	}
+}
+
+func bucketPackages(timing map[string]time.Duration, packages []string, n uint) []bucket {
+	sort.SliceStable(packages, func(i, j int) bool {
+		return timing[packages[i]] >= timing[packages[j]]
+	})
+
+	buckets := make([]bucket, n)
+	for _, pkg := range packages {
+		i := minBucket(buckets)
+		buckets[i].Total += timing[pkg]
+		buckets[i].Packages = append(buckets[i].Packages, pkg)
+		log.Debugf("adding %v (%v) to bucket %v with total %v",
+			pkg, timing[pkg], i, buckets[i].Total)
+	}
+	return buckets
+}
+
+func minBucket(buckets []bucket) int {
+	var n int
+	var min time.Duration = -1
+	for i, b := range buckets {
+		switch {
+		case min < 0 || b.Total < min:
+			min = b.Total
+			n = i
+		case b.Total == min && len(buckets[i].Packages) < len(buckets[n].Packages):
+			n = i
+		}
+	}
+	return n
+}
+
+type bucket struct {
+	Total    time.Duration
+	Packages []string
+}
+
+type matrix struct {
+	Include []Partition `json:"include"`
+}
+
+type Partition struct {
+	ID               int    `json:"id"`
+	EstimatedRuntime string `json:"estimatedRuntime"`
+	Packages         string `json:"packages"`
+	Description      string `json:"description"`
+}
+
+func writeMatrix(out io.Writer, buckets []bucket) error {
+	m := matrix{Include: make([]Partition, len(buckets))}
+	for i, bucket := range buckets {
+		p := Partition{
+			ID:               i,
+			EstimatedRuntime: bucket.Total.String(),
+			Packages:         strings.Join(bucket.Packages, " "),
+		}
+		if len(bucket.Packages) > 0 {
+			var extra string
+			if len(bucket.Packages) > 1 {
+				extra = fmt.Sprintf(" and %d others", len(bucket.Packages)-1)
+			}
+			p.Description = fmt.Sprintf("partition %d - package %v%v",
+				p.ID, testjson.RelativePackagePath(bucket.Packages[0]), extra)
+		}
+
+		m.Include[i] = p
+	}
+
+	log.Debugf("%v\n", debugMatrix(m))
+
+	err := json.NewEncoder(out).Encode(m)
+	if err != nil {
+		return fmt.Errorf("failed to json encode output: %v", err)
+	}
+	return nil
+}
+
+type debugMatrix matrix
+
+func (d debugMatrix) String() string {
+	raw, err := json.MarshalIndent(d, "", "  ")
+	if err != nil {
+		return fmt.Sprintf("failed to marshal: %v", err.Error())
+	}
+	return string(raw)
+}
diff --git a/internal/log/log.go b/internal/log/log.go
index 9a3e85f..d68955b 100644
--- a/internal/log/log.go
+++ b/internal/log/log.go
@@ -11,6 +11,7 @@ type Level uint8
 const (
 	ErrorLevel Level = iota
 	WarnLevel
+	InfoLevel
 	DebugLevel
 )
 
@@ -43,6 +44,15 @@ func Debugf(format string, args ...interface{}) {
 	fmt.Fprint(out, "\n")
 }
 
+// Infof prints the message to stderr, with no prefix.
+func Infof(format string, args ...interface{}) {
+	if level < InfoLevel {
+		return
+	}
+	fmt.Fprintf(out, format, args...)
+	fmt.Fprint(out, "\n")
+}
+
 // Errorf prints the message to stderr, with a red ERROR prefix.
 func Errorf(format string, args ...interface{}) {
 	if level < ErrorLevel {
diff --git a/main.go b/main.go
index d2ed49f..f8af75e 100644
--- a/main.go
+++ b/main.go
@@ -5,6 +5,7 @@ import (
 	"os"
 
 	"gotest.tools/gotestsum/cmd"
+	"gotest.tools/gotestsum/cmd/tool/matrix"
 	"gotest.tools/gotestsum/cmd/tool/slowest"
 	"gotest.tools/gotestsum/internal/log"
 )
@@ -55,6 +56,7 @@ func toolRun(name string, args []string) error {
 
 Commands:
     %[1]s slowest      find or skip the slowest tests
+    %[1]s ci-matrix    use previous test runtime to place packages into optimal buckets
 
 Use '%[1]s COMMAND --help' for command specific help.
 `, name)
@@ -67,6 +69,8 @@ Use '%[1]s COMMAND --help' for command specific help.
 		return nil
 	case "slowest":
 		return slowest.Run(name+" "+next, rest)
+	case "ci-matrix":
+		return matrix.Run(name+" "+next, rest)
 	default:
 		fmt.Fprintln(os.Stderr, usage(name))
 		return fmt.Errorf("invalid command: %v %v", name, next)


Test patch:
diff --git a/cmd/tool/matrix/matrix_test.go b/cmd/tool/matrix/matrix_test.go
new file mode 100644
index 0000000..b537318
--- /dev/null
+++ b/cmd/tool/matrix/matrix_test.go
@@ -0,0 +1,258 @@
+package matrix
+
+import (
+	"bytes"
+	"encoding/json"
+	"io"
+	"os"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"gotest.tools/gotestsum/testjson"
+	"gotest.tools/v3/assert"
+	"gotest.tools/v3/fs"
+)
+
+func TestPackagePercentile(t *testing.T) {
+	ms := time.Millisecond
+	timing := map[string][]time.Duration{
+		"none":  {},
+		"one":   {time.Second},
+		"two":   {4 * ms, 2 * ms},
+		"three": {2 * ms, 3 * ms, 5 * ms},
+		"four":  {4 * ms, 3 * ms, ms, 2 * ms},
+		"five":  {6 * ms, 2 * ms, 3 * ms, 4 * ms, 9 * ms},
+		"nine":  {6 * ms, 2 * ms, 3 * ms, 4 * ms, 9 * ms, 1 * ms, 5 * ms, 7 * ms, 8 * ms},
+		"ten":   {6 * ms, 2 * ms, 3 * ms, 4 * ms, 9 * ms, 5 * ms, 7 * ms, 8 * ms, ms, ms},
+		"twenty": {
+			6 * ms, 2 * ms, 3 * ms, 4 * ms, 9 * ms, 5 * ms, 7 * ms, 8 * ms, ms, ms,
+			100, 200, 600, 700, 800, 900, 200, 300, 400, 500,
+		},
+	}
+
+	out := packagePercentile(timing)
+	expected := map[string]time.Duration{
+		"none":   0,
+		"one":    time.Second,
+		"two":    4 * ms,
+		"three":  5 * ms,
+		"four":   4 * ms,
+		"five":   9 * ms,
+		"nine":   8 * ms,
+		"ten":    8 * ms,
+		"twenty": 6 * ms,
+	}
+	assert.DeepEqual(t, out, expected)
+}
+
+func TestBucketPackages(t *testing.T) {
+	ms := time.Millisecond
+	timing := map[string]time.Duration{
+		"one":   190 * ms,
+		"two":   200 * ms,
+		"three": 3800 * ms,
+		"four":  4000 * ms,
+		"five":  50 * ms,
+		"six":   606 * ms,
+		"rm1":   time.Second,
+		"rm2":   time.Second,
+	}
+	packages := []string{"new1", "new2", "one", "two", "three", "four", "five", "six"}
+
+	type testCase struct {
+		n        uint
+		expected []bucket
+	}
+
+	run := func(t *testing.T, tc testCase) {
+		buckets := bucketPackages(timing, packages, tc.n)
+		assert.DeepEqual(t, buckets, tc.expected)
+	}
+
+	testCases := []testCase{
+		{
+			n: 2,
+			expected: []bucket{
+				0: {Total: 4440 * ms, Packages: []string{"four", "two", "one", "five"}},
+				1: {Total: 4406 * ms, Packages: []string{"three", "six", "new2", "new1"}},
+			},
+		},
+		{
+			n: 3,
+			expected: []bucket{
+				0: {Total: 4000 * ms, Packages: []string{"four"}},
+				1: {Total: 3800 * ms, Packages: []string{"three"}},
+				2: {Total: 1046 * ms, Packages: []string{"six", "two", "one", "five", "new1", "new2"}},
+			},
+		},
+		{
+			n: 4,
+			expected: []bucket{
+				0: {Total: 4000 * ms, Packages: []string{"four"}},
+				1: {Total: 3800 * ms, Packages: []string{"three"}},
+				2: {Total: 606 * ms, Packages: []string{"six"}},
+				3: {Total: 440 * ms, Packages: []string{"two", "one", "five", "new2", "new1"}},
+			},
+		},
+		{
+			n: 8,
+			expected: []bucket{
+				0: {Total: 4000 * ms, Packages: []string{"four"}},
+				1: {Total: 3800 * ms, Packages: []string{"three"}},
+				2: {Total: 606 * ms, Packages: []string{"six"}},
+				3: {Total: 200 * ms, Packages: []string{"two"}},
+				4: {Total: 190 * ms, Packages: []string{"one"}},
+				5: {Total: 50 * ms, Packages: []string{"five"}},
+				6: {Packages: []string{"new1"}},
+				7: {Packages: []string{"new2"}},
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(strconv.FormatUint(uint64(tc.n), 10), func(t *testing.T) {
+			run(t, tc)
+		})
+	}
+}
+
+func TestReadTimingReports(t *testing.T) {
+	events := func(t *testing.T, start time.Time) string {
+		t.Helper()
+		var buf bytes.Buffer
+		encoder := json.NewEncoder(&buf)
+		for _, i := range []int{0, 1, 2} {
+			assert.NilError(t, encoder.Encode(testjson.TestEvent{
+				Time:    start.Add(time.Duration(i) * time.Second),
+				Action:  testjson.ActionRun,
+				Package: "pkg" + strconv.Itoa(i),
+			}))
+		}
+		return buf.String()
+	}
+
+	now := time.Now()
+	dir := fs.NewDir(t, "timing-files",
+		fs.WithFile("report1.log", events(t, now.Add(-time.Hour))),
+		fs.WithFile("report2.log", events(t, now.Add(-47*time.Hour))),
+		fs.WithFile("report3.log", events(t, now.Add(-49*time.Hour))),
+		fs.WithFile("report4.log", events(t, now.Add(-101*time.Hour))))
+
+	t.Run("match files", func(t *testing.T) {
+		opts := options{
+			timingFilesPattern: dir.Join("*.log"),
+		}
+
+		files, err := readTimingReports(opts)
+		assert.NilError(t, err)
+		defer closeFiles(files)
+		assert.Equal(t, len(files), 4)
+
+		for _, fh := range files {
+			// check the files are properly seeked to 0
+			event, err := parseEvent(fh)
+			assert.NilError(t, err)
+			assert.Equal(t, event.Package, "pkg0")
+		}
+
+		actual, err := os.ReadDir(dir.Path())
+		assert.NilError(t, err)
+		assert.Equal(t, len(actual), 4)
+	})
+
+	t.Run("no glob match, func", func(t *testing.T) {
+		opts := options{
+			timingFilesPattern: dir.Join("*.json"),
+		}
+
+		files, err := readTimingReports(opts)
+		assert.NilError(t, err)
+		assert.Equal(t, len(files), 0)
+	})
+}
+
+func TestRun(t *testing.T) {
+	events := func(t *testing.T) string {
+		t.Helper()
+		var buf bytes.Buffer
+		encoder := json.NewEncoder(&buf)
+		for _, i := range []int{0, 1, 2} {
+			elapsed := time.Duration(i+1) * 2 * time.Second
+			end := time.Now().Add(-5 * time.Second)
+			start := end.Add(-elapsed)
+
+			assert.NilError(t, encoder.Encode(testjson.TestEvent{
+				Time:    start,
+				Action:  testjson.ActionRun,
+				Package: "pkg" + strconv.Itoa(i),
+			}))
+			assert.NilError(t, encoder.Encode(testjson.TestEvent{
+				Time:    end,
+				Action:  testjson.ActionPass,
+				Package: "pkg" + strconv.Itoa(i),
+				Elapsed: elapsed.Seconds(),
+			}))
+		}
+		return buf.String()
+	}
+
+	dir := fs.NewDir(t, "timing-files",
+		fs.WithFile("report1.log", events(t)),
+		fs.WithFile("report2.log", events(t)),
+		fs.WithFile("report3.log", events(t)),
+		fs.WithFile("report4.log", events(t)),
+		fs.WithFile("report5.log", events(t)))
+
+	stdout := new(bytes.Buffer)
+	opts := options{
+		numPartitions:      3,
+		timingFilesPattern: dir.Join("*.log"),
+		debug:              true,
+		stdout:             stdout,
+		stdin:              strings.NewReader("pkg0\npkg1\npkg2\nother"),
+	}
+	err := run(opts)
+	assert.NilError(t, err)
+	assert.Equal(t, strings.Count(stdout.String(), "\n"), 1,
+		"the output should be a single line")
+
+	assert.Equal(t, formatJSON(t, stdout), expectedMatrix)
+}
+
+// expectedMatrix can be automatically updated by running tests with -update
+// nolint:lll
+var expectedMatrix = `{
+  "include": [
+    {
+      "description": "partition 0 - package pkg2",
+      "estimatedRuntime": "6s",
+      "id": 0,
+      "packages": "pkg2"
+    },
+    {
+      "description": "partition 1 - package pkg1",
+      "estimatedRuntime": "4s",
+      "id": 1,
+      "packages": "pkg1"
+    },
+    {
+      "description": "partition 2 - package pkg0 and 1 others",
+      "estimatedRuntime": "2s",
+      "id": 2,
+      "packages": "pkg0 other"
+    }
+  ]
+}`
+
+func formatJSON(t *testing.T, v io.Reader) string {
+	t.Helper()
+	raw := map[string]interface{}{}
+	err := json.NewDecoder(v).Decode(&raw)
+	assert.NilError(t, err)
+
+	formatted, err := json.MarshalIndent(raw, "", "  ")
+	assert.NilError(t, err)
+	return string(formatted)
+}