Repository: exercism/cli
Pull request #1092 (2023-06-25 05:37:20)

Add `test` command to run any unit test
Per discussion in [this forum post](https://forum.exercism.org/t/introducing-the-universal-test-runner/6228), I've added a `test` command to the exercism CLI. Much like the [universal test runner](https://github.com/xavdid/universal-test-runner) it's inspired by, its goal is to be able to run the basic unit test suite for any exercism exercise. 

It does this by:

1. reading `.exercism/metadata.json` to get the track name
2. doing a lookup in a table which maps track name => a test command (and some configuration; see below)
3. maybe reading `.exercism/config.json` to get test file name(s) and including them
4. including any args from `exercism test`
5. Using go's `exec.Command` to run and return info about the exit code

## Types of Tests

Based on my (very brief) sampling of exercism tracks, there are 2 main ways tests are run:

1. Using a language-wide common command (e.g `cargo test`, `go test`, `lien test`, etc)
2. A single test file invoked like any program of that language (e.g. `ruby lasagna_test.rb`)

So, the `TestConfiguration` struct covers both of those cases (by including an option to append the test file(s) to the command).

The other configuration option aids in arg passing. Some language tools (namely `cargo`) expect arguments to the test command to be after a double dash (`--`). Unfortunately, so does `Cobra,` the `go` arg parser. So, I was running into the following trying to run rust tests with `--include-ignored`:

1. `exercism test --include-ignored`: `--include-ignored` is not a valid flag for the test command
2. `exercism test -- --include-ignored`: `--include-ignored` is not a valid flag for `cargo`, put it behind a `--` 
3. `exercism test -- -- --include-ignored`: works as expected

This isn't great UX though, so I added an option for languages to be able to ask that args to their test runner be put behind an implied `--`. This way, the rust command works when run as option 2 above (which is in line with most other languages).

## TODO

- [x] add tests
- [x] add more languages (I don't know that we need to ship with 100% coverage, but the top 20 most popular languages would be great)
- [x] add docs / changelog

github-actions[bot]: Hello. Thanks for opening a PR on Exercism. We are currently in a phase of our journey where we have paused community contributions to allow us to take a breather and redesign our community model. You can learn more in [this blog post](https://exercism.org/blog/freeing-our-maintainers). **As such, all issues and PRs in this repository are being automatically closed.**

That doesn't mean we're not interested in your ideas, or that if you're stuck on something we don't want to help. The best place to discuss things is with our community on the Exercism Community Forum. You can use [this link](https://forum.exercism.org/new-topic?title=Add%20%60test%60%20command%20to%20run%20any%20unit%20test&body=Per%20discussion%20in%20%5Bthis%20forum%20post%5D(https://forum.exercism.org/t/introducing-the-universal-test-runner/6228),%20I've%20added%20a%20%60test%60%20command%20to%20the%20exercism%20CLI.%20Much%20like%20the%20%5Buniversal%20test%20runner%5D(https://github.com/xavdid/universal-test-runner)%20it's%20inspired%20by,%20its%20goal%20is%20to%20be%20able%20to%20run%20the%20basic%20unit%20test%20suite%20for%20any%20exercism%20exercise.%20%0D%0A%0D%0AIt%20does%20this%20by:%0D%0A%0D%0A1.%20reading%20%60.exercism/metadata.json%60%20to%20get%20the%20track%20name%0D%0A2.%20doing%20a%20lookup%20in%20a%20table%20which%20maps%20track%20name%20=%3E%20a%20test%20command%20(and%20some%20configuration;%20see%20below)%0D%0A3.%20maybe%20reading%20%60.exercism/config.json%60%20to%20get%20test%20file%20name(s)%20and%20including%20them%0D%0A4.%20including%20any%20args%20from%20%60exercism%20test%60%0D%0A5.%20Using%20go's%20%60exec.Command%60%20to%20run%20and%20return%20info%20about%20the%20exit%20code%0D%0A%0D%0A##%20Types%20of%20Tests%0D%0A%0D%0ABased%20on%20my%20(very%20brief)%20sampling%20of%20exercism%20tracks,%20there%20are%202%20main%20ways%20tests%20are%20run:%0D%0A%0D%0A1.%20Using%20a%20language-wide%20common%20command%20(e.g%20%60cargo%20test%60,%20%60go%20test%60,%20%60lien%20test%60,%20etc)%0D%0A2.%20A%20single%20test%20file%20invoked%20like%20any%20program%20of%20that%20language%20(e.g.%20%60ruby%20lasagna_test.rb%60)%0D%0A%0D%0ASo,%20the%20%60TestConfiguration%60%20struct%20covers%20both%20of%20those%20cases%20(by%20including%20an%20option%20to%20append%20the%20test%20file(s)%20to%20the%20command).%0D%0A%0D%0AThe%20other%20configuration%20option%20aids%20in%20arg%20passing.%20Some%20language%20tools%20(namely%20%60cargo%60)%20expect%20arguments%20to%20the%20test%20command%20to%20be%20after%20a%20double%20dash%20(%60--%60).%20Unfortunately,%20so%20does%20%60Cobra,%60%20the%20%60go%60%20arg%20parser.%20So,%20I%20was%20running%20into%20the%20following%20trying%20to%20run%20rust%20tests%20with%20%60--include-ignored%60:%0D%0A%0D%0A1.%20%60exercism%20test%20--include-ignored%60:%20%60--include-ignored%60%20is%20not%20a%20valid%20flag%20for%20the%20test%20command%0D%0A2.%20%60exercism%20test%20--%20--include-ignored%60:%20%60--include-ignored%60%20is%20not%20a%20valid%20flag%20for%20%60cargo%60,%20put%20it%20behind%20a%20%60--%60%20%0D%0A3.%20%60exercism%20test%20--%20--%20--include-ignored%60:%20works%20as%20expected%0D%0A%0D%0AThis%20isn't%20great%20UX%20though,%20so%20I%20added%20an%20option%20for%20languages%20to%20be%20able%20to%20ask%20that%20args%20to%20their%20test%20runner%20be%20put%20behind%20an%20implied%20%60--%60.%20This%20way,%20the%20rust%20command%20works%20when%20run%20as%20option%202%20above%20(which%20is%20in%20line%20with%20most%20other%20languages).%0D%0A%0D%0A##%20TODO%0D%0A%0D%0A-%20%5B%20%5D%20add%20tests%0D%0A-%20%5B%20%5D%20add%20more%20languages%20(I%20don't%20know%20that%20we%20need%20to%20ship%20with%20100%25%20coverage,%20but%20the%20top%2020%20most%20popular%20languages%20would%20be%20great)%0D%0A-%20%5B%20%5D%20add%20docs%20/%20changelog&category=support) to copy this into a new topic there.

---

_Note: If this PR has been pre-approved, please link back to this PR on the forum thread and a maintainer or staff member will reopen it._

ErikSchierboom: I'd like someone with Go experience to look at this (@ekingery or @junedev perhaps?).

I'm doing some functional testing:

- Simple command (tested with C#): works
- Command with {{test_files}} placeholder (tested with CoffeeScript and Ruby): work
- Windows/Linux commands: I don't have Go on my Windows machine, but the Linux bit works. Someone else should test this on Windows
junedev: @ErikSchierboom The Go code itself looks fine to me. Can't comment on general code organization in the codebase or anything like that though.

Sidenote: On a conceptual level, I am not sure whether I am a big fan of this feature as we take away learning about how to actually run tests in the language. Even if it means just learning that in this language usually a make file is used or "there is a built-in test runner" like in the case of Go. I am ok with this being merged but I wanted to mention it.
xavdid: Ok, I've added little unit tests for the exercise config reading. It doesn't look like it's easy to test the windows behavior in a unit test (based on [this SO answer](https://stackoverflow.com/questions/59230895/how-should-i-handle-testing-of-a-go-function-that-uses-runtime-goos)), but I have a windows computer I can probably install `go` on and try it out.

> I am not sure whether I am a big fan of this feature as we take away learning about how to actually run tests in the language.

That's a great point! I got similar feedback on the standalone project. I've added a logline that shows exactly what's being run, so at least it's more clear what's happening:

```
% ~/projects/exercism-cli/testercism test
--> ruby lasagna_test.rb
Run options: --seed 33076

# Running:
....
```

If we're comfortable (enough to move forward) with the level of unit tests written, I think the last big tasks is go through the rest of the "easy" and "medium" language tracks and add them into the test configurations. I've been saving that for last so the PR doesn't get too unwieldy, but can grab it tomorrow. 
xavdid: Ok, I've added more tests than I thought I'd be able to. I saw that CI actually has a windows runner, so I added a test that pulls the windows command. I also got some file-based tests working by actually writing and removing files. It's in the test directory, which I don't love, but it should be fine.

The last thing is the actual `test` command itself, but that would require more refactoring to mock out the `exec.Command` call. The rest of it has enough tests that I'm personally comfortable shipping this PR as stands (+ the rest of the easy commands, as mentioned; I'll add those after the rest is ✅)

Thanks for bearing with me y'all!
ErikSchierboom: @ekingery Are you up for a code review? I'm especially interested in the test coverage.
ErikSchierboom: Thanks for the reviews everybody!
ErikSchierboom: Really happy about the state of this PR!

There are two things left to do:

1. Fix Windows CI
2. Add the test configurations for all tracks
xavdid: Ah, pesky platform-specific error message. I _think_ I fixed CI in 540bef5. I'll get the rest of the tests in tomorrow. 
xavdid: Ok, I've added the remaining tracks (and more than I thought I'd get)!

Work is split into a few last commits:

- I added all the simple tests in https://github.com/exercism/cli/pull/1092/commits/fc6b22cc2a99db53d6a9e9d09f9f00340c21713f
- added `bash`-based tests and some sensible defaults for ambiguous tracks (`d` and `clojure`) in https://github.com/exercism/cli/pull/1092/commits/dcdf1681767fb079c053d0c9ab0aa3ef066665c7
- added `(c *ExerciseConfig) GetSolutionFiles()` and its tests (which closely mirrors `GetTestFiles()`) to power the `prolog` track (which needs both solution and test files in the command) in https://github.com/exercism/cli/pull/1092/commits/dcdf1681767fb079c053d0c9ab0aa3ef066665c7
- updated the changelog in https://github.com/exercism/cli/pull/1092/commits/8d18c398e3d2f51cb86d90945fd7febc03a8ad2b

@ErikSchierboom unless anything from those jumps out at you, I think we're all set!

The only remaining tracks are `scheme` (two tools mentioned in docs, unclear which a user will likely have installed) and `cpp` + `fortran` (complex, multi-stage setup process). My full notes are [in Airtable](https://airtable.com/appN6G2B1ylN5OQJa/shrIkepZbMogNSFpW) if you need to refer back to them.
ErikSchierboom: > @ErikSchierboom unless anything from those jumps out at you, I think we're all set!

I'm quite busy today, so it will be next week (wednesday at the earliest). Sorry!
xavdid: Ok, I debugged the tests on my windows machine and got them working in https://github.com/exercism/cli/pull/1092/commits/238b5729850c09a2ebc2ad0a7e1bef9895f6815d. Turns out windows won't delete a folder if it's got an open descriptor somewhere and I had neglected to close after write. Mac had no issue with this, so it was news to me. Also, I was using `path.join` instead of `filepath.join` in a couple of tests.

That CI _should_ be good to go. 🤞 
xavdid: Ok, addressed that feedback. CI should be good to go as well!
ErikSchierboom: ```shell
erik@schierheim:~/exercism/cli$ git log --all --full-history -- testercism
erik@schierheim:~/exercism/cli$ 
```

Nice! That looks good. Once CI is green, I'll merge. Thank you for this great contribution!
xavdid: All right! Looks like we're all set. 

You're very welcome, glad we got it over the line. Thank you for your patience with the whole process - this one was a journey 😅 

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 110c6fe..1c84fa0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@ The exercism CLI follows [semantic versioning](http://semver.org/).
 ----------------
 
 ## Next Release
+* [#1092](https://github.com/exercism/cli/pull/1092) Add `exercism test` command to run the unit tests for nearly any track (inspired by [universal-test-runner](https://github.com/xavdid/universal-test-runner)) - [@xavdid]
 * **Your contribution here**
 
 ## v3.1.0 (2022-10-04)
@@ -489,5 +490,6 @@ All changes by [@msgehard]
 [@sfairchild]: https://github.com/sfairchild
 [@simonjefford]: https://github.com/simonjefford
 [@srt32]: https://github.com/srt32
+[@xavdid]: https://github.com/xavdid
 [@williandrade]: https://github.com/williandrade
 [@zabawaba99]: https://github.com/zabawaba99
diff --git a/workspace/exercise_config.go b/workspace/exercise_config.go
new file mode 100644
index 0000000..9b0164e
--- /dev/null
+++ b/workspace/exercise_config.go
@@ -0,0 +1,57 @@
+package workspace
+
+import (
+	"encoding/json"
+	"errors"
+	"io/ioutil"
+	"path/filepath"
+)
+
+const configFilename = "config.json"
+
+var configFilepath = filepath.Join(ignoreSubdir, configFilename)
+
+// ExerciseConfig contains exercise metadata.
+// Note: we only use a subset of its fields
+type ExerciseConfig struct {
+	Files struct {
+		Solution []string `json:"solution"`
+		Test     []string `json:"test"`
+	} `json:"files"`
+}
+
+// NewExerciseConfig reads exercise metadata from a file in the given directory.
+func NewExerciseConfig(dir string) (*ExerciseConfig, error) {
+	b, err := ioutil.ReadFile(filepath.Join(dir, configFilepath))
+	if err != nil {
+		return nil, err
+	}
+	var config ExerciseConfig
+	if err := json.Unmarshal(b, &config); err != nil {
+		return nil, err
+	}
+
+	return &config, nil
+}
+
+// GetTestFiles finds returns the names of the file(s) that hold unit tests for this exercise, if any
+func (c *ExerciseConfig) GetSolutionFiles() ([]string, error) {
+	result := c.Files.Solution
+	if result == nil {
+		// solution file(s) key was missing in config json, which is an error when calling this fuction
+		return []string{}, errors.New("no `files.solution` key in your `config.json`. Was it removed by mistake?")
+	}
+
+	return result, nil
+}
+
+// GetTestFiles finds returns the names of the file(s) that hold unit tests for this exercise, if any
+func (c *ExerciseConfig) GetTestFiles() ([]string, error) {
+	result := c.Files.Test
+	if result == nil {
+		// test file(s) key was missing in config json, which is an error when calling this fuction
+		return []string{}, errors.New("no `files.test` key in your `config.json`. Was it removed by mistake?")
+	}
+
+	return result, nil
+}


Test patch:
diff --git a/cmd/cmd_test.go b/cmd/cmd_test.go
index 76ed063..fee08cd 100644
--- a/cmd/cmd_test.go
+++ b/cmd/cmd_test.go
@@ -26,12 +26,14 @@ const cfgHomeKey = "EXERCISM_CONFIG_HOME"
 // test, call the command by calling Execute on the App.
 //
 // Example:
-// cmdTest := &CommandTest{
-// 	Cmd:    myCmd,
-// 	InitFn: initMyCmd,
-// 	Args:   []string{"fakeapp", "mycommand", "arg1", "--flag", "value"},
-// 	MockInteractiveResponse: "first-input\nsecond\n",
-// }
+//
+//	cmdTest := &CommandTest{
+//		Cmd:    myCmd,
+//		InitFn: initMyCmd,
+//		Args:   []string{"fakeapp", "mycommand", "arg1", "--flag", "value"},
+//		MockInteractiveResponse: "first-input\nsecond\n",
+//	}
+//
 // cmdTest.Setup(t)
 // defer cmdTest.Teardown(t)
 // ...
diff --git a/cmd/test.go b/cmd/test.go
new file mode 100644
index 0000000..8f5b54e
--- /dev/null
+++ b/cmd/test.go
@@ -0,0 +1,84 @@
+package cmd
+
+import (
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"strings"
+
+	"github.com/exercism/cli/workspace"
+	"github.com/spf13/cobra"
+)
+
+var testCmd = &cobra.Command{
+	Use:     "test",
+	Aliases: []string{"t"},
+	Short:   "Run the exercise's tests.",
+	Long: `Run the exercise's tests.
+
+	Run this command in an exercise's root directory.`,
+	RunE: func(cmd *cobra.Command, args []string) error {
+		return runTest(args)
+	},
+}
+
+func runTest(args []string) error {
+	track, err := getTrack()
+	if err != nil {
+		return err
+	}
+
+	testConf, ok := workspace.TestConfigurations[track]
+
+	if !ok {
+		return fmt.Errorf("the \"%s\" track does not yet support running tests using the Exercism CLI. Please see HELP.md for testing instructions", track)
+	}
+
+	command, err := testConf.GetTestCommand()
+	if err != nil {
+		return err
+	}
+	cmdParts := strings.Split(command, " ")
+
+	// pass args/flags to this command down to the test handler
+	if len(args) > 0 {
+		cmdParts = append(cmdParts, args...)
+	}
+
+	fmt.Printf("Running tests via `%s`\n\n", strings.Join(cmdParts, " "))
+	exerciseTestCmd := exec.Command(cmdParts[0], cmdParts[1:]...)
+
+	// pipe output directly out, preserving any color
+	exerciseTestCmd.Stdout = os.Stdout
+	exerciseTestCmd.Stderr = os.Stderr
+
+	err = exerciseTestCmd.Run()
+	if err != nil {
+		// unclear what other errors would pop up here, but it pays to be defensive
+		if exitErr, ok := err.(*exec.ExitError); ok {
+			exitCode := exitErr.ExitCode()
+			// if subcommand returned a non-zero exit code, exit with the same
+			os.Exit(exitCode)
+		} else {
+			log.Fatalf("Failed to get error from failed subcommand: %v", err)
+		}
+	}
+	return nil
+}
+
+func getTrack() (string, error) {
+	metadata, err := workspace.NewExerciseMetadata(".")
+	if err != nil {
+		return "", err
+	}
+	if metadata.Track == "" {
+		return "", fmt.Errorf("no track found in exercise metadata")
+	}
+
+	return metadata.Track, nil
+}
+
+func init() {
+	RootCmd.AddCommand(testCmd)
+}
diff --git a/workspace/exercise_config_test.go b/workspace/exercise_config_test.go
new file mode 100644
index 0000000..d07264c
--- /dev/null
+++ b/workspace/exercise_config_test.go
@@ -0,0 +1,98 @@
+package workspace
+
+import (
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestExerciseConfig(t *testing.T) {
+	dir, err := ioutil.TempDir("", "exercise_config")
+	assert.NoError(t, err)
+	defer os.RemoveAll(dir)
+
+	err = os.Mkdir(filepath.Join(dir, ".exercism"), os.ModePerm)
+	assert.NoError(t, err)
+
+	f, err := os.Create(filepath.Join(dir, ".exercism", "config.json"))
+	assert.NoError(t, err)
+	defer f.Close()
+
+	_, err = f.WriteString(`{ "blurb": "Learn about the basics of Ruby by following a lasagna recipe.", "authors": ["iHiD", "pvcarrera"], "files": { "solution": ["lasagna.rb"], "test": ["lasagna_test.rb"], "exemplar": [".meta/exemplar.rb"] } } `)
+	assert.NoError(t, err)
+
+	ec, err := NewExerciseConfig(dir)
+	assert.NoError(t, err)
+
+	assert.Equal(t, ec.Files.Solution, []string{"lasagna.rb"})
+	solutionFiles, err := ec.GetSolutionFiles()
+	assert.NoError(t, err)
+	assert.Equal(t, solutionFiles, []string{"lasagna.rb"})
+
+	assert.Equal(t, ec.Files.Test, []string{"lasagna_test.rb"})
+	testFiles, err := ec.GetTestFiles()
+	assert.NoError(t, err)
+	assert.Equal(t, testFiles, []string{"lasagna_test.rb"})
+}
+
+func TestExerciseConfigNoTestKey(t *testing.T) {
+	dir, err := ioutil.TempDir("", "exercise_config")
+	assert.NoError(t, err)
+	defer os.RemoveAll(dir)
+
+	err = os.Mkdir(filepath.Join(dir, ".exercism"), os.ModePerm)
+	assert.NoError(t, err)
+
+	f, err := os.Create(filepath.Join(dir, ".exercism", "config.json"))
+	assert.NoError(t, err)
+	defer f.Close()
+
+	_, err = f.WriteString(`{ "blurb": "Learn about the basics of Ruby by following a lasagna recipe.", "authors": ["iHiD", "pvcarrera"], "files": { "exemplar": [".meta/exemplar.rb"] } } `)
+	assert.NoError(t, err)
+
+	ec, err := NewExerciseConfig(dir)
+	assert.NoError(t, err)
+
+	_, err = ec.GetSolutionFiles()
+	assert.Error(t, err, "no `files.solution` key in your `config.json`")
+	_, err = ec.GetTestFiles()
+	assert.Error(t, err, "no `files.test` key in your `config.json`")
+}
+
+func TestMissingExerciseConfig(t *testing.T) {
+	dir, err := ioutil.TempDir("", "exercise_config")
+	assert.NoError(t, err)
+	defer os.RemoveAll(dir)
+
+	_, err = NewExerciseConfig(dir)
+	assert.Error(t, err)
+	// any assertions about this error message have to work across all platforms, so be vague
+	// unix: ".exercism/config.json: no such file or directory"
+	// windows: "open .exercism\config.json: The system cannot find the path specified."
+	assert.Contains(t, err.Error(), filepath.Join(".exercism", "config.json:"))
+}
+
+func TestInvalidExerciseConfig(t *testing.T) {
+	dir, err := ioutil.TempDir("", "exercise_config")
+	assert.NoError(t, err)
+	defer os.RemoveAll(dir)
+
+	err = os.Mkdir(filepath.Join(dir, ".exercism"), os.ModePerm)
+	assert.NoError(t, err)
+
+	f, err := os.Create(filepath.Join(dir, ".exercism", "config.json"))
+	assert.NoError(t, err)
+	defer f.Close()
+
+	// invalid JSON
+	_, err = f.WriteString(`{ "blurb": "Learn about the basics of Ruby by following a lasagna recipe.", "authors": ["iHiD", "pvcarr `)
+	assert.NoError(t, err)
+
+	_, err = NewExerciseConfig(dir)
+	assert.Error(t, err)
+	assert.True(t, strings.Contains(err.Error(), "unexpected end of JSON input"))
+}
diff --git a/workspace/test_configurations.go b/workspace/test_configurations.go
new file mode 100644
index 0000000..32c6fbc
--- /dev/null
+++ b/workspace/test_configurations.go
@@ -0,0 +1,252 @@
+package workspace
+
+import (
+	"fmt"
+	"runtime"
+	"strings"
+)
+
+type TestConfiguration struct {
+	// The static portion of the test Command, which will be run for every test on this track. Examples include `cargo test` or `go test`.
+	// Might be empty if there are platform-specific versions
+	Command string
+
+	// Windows-specific test command. Mostly relevant for tests wrapped by shell invocations. Falls back to `Command` if we're not running windows or this is empty.
+	WindowsCommand string
+}
+
+func (c *TestConfiguration) GetTestCommand() (string, error) {
+	var cmd string
+	if runtime.GOOS == "windows" && c.WindowsCommand != "" {
+		cmd = c.WindowsCommand
+	} else {
+		cmd = c.Command
+	}
+
+	// pre-declare these so we can conditionally initialize them
+	var exerciseConfig *ExerciseConfig
+	var err error
+
+	if strings.Contains(cmd, "{{") {
+		// only read exercise's config.json if we need it
+		exerciseConfig, err = NewExerciseConfig(".")
+		if err != nil {
+			return "", err
+		}
+	}
+
+	if strings.Contains(cmd, "{{solution_files}}") {
+		if exerciseConfig == nil {
+			return "", fmt.Errorf("exerciseConfig not initialize before use")
+		}
+		solutionFiles, err := exerciseConfig.GetSolutionFiles()
+		if err != nil {
+			return "", err
+		}
+		cmd = strings.ReplaceAll(cmd, "{{solution_files}}", strings.Join(solutionFiles, " "))
+	}
+	if strings.Contains(cmd, "{{test_files}}") {
+		if exerciseConfig == nil {
+			return "", fmt.Errorf("exerciseConfig not initialize before use")
+		}
+		testFiles, err := exerciseConfig.GetTestFiles()
+		if err != nil {
+			return "", err
+		}
+		cmd = strings.ReplaceAll(cmd, "{{test_files}}", strings.Join(testFiles, " "))
+	}
+
+	return cmd, nil
+}
+
+// some tracks aren't (or won't be) implemented; every track is listed either way
+var TestConfigurations = map[string]TestConfiguration{
+	"8th": {
+		Command:        "bash tester.sh",
+		WindowsCommand: "tester.bat",
+	},
+	// abap: tests are run via "ABAP Development Tools", not the CLI
+	"awk": {
+		Command: "bats {{test_files}}",
+	},
+	"ballerina": {
+		Command: "bal test",
+	},
+	"bash": {
+		Command: "bats {{test_files}}",
+	},
+	"c": {
+		Command: "make",
+	},
+	"cfml": {
+		Command: "box task run TestRunner",
+	},
+	"clojure": {
+		// chosen because the docs recommend `clj` by default and `lein` as optional
+		Command: "clj -X:test",
+	},
+	"cobol": {
+		Command:        "bash test.sh",
+		WindowsCommand: "pwsh test.ps1",
+	},
+	"coffeescript": {
+		Command: "jasmine-node --coffee {{test_files}}",
+	},
+	// common-lisp: tests are loaded into a "running Lisp implementation", not the CLI directly
+	"cpp": {
+		Command: "make",
+	},
+	"crystal": {
+		Command: "crystal spec",
+	},
+	"csharp": {
+		Command: "dotnet test",
+	},
+	"d": {
+		// this always works even if the user installed DUB
+		Command: "dmd source/*.d -de -w -main -unittest",
+	},
+	"dart": {
+		Command: "dart test",
+	},
+	// delphi: tests are run via IDE
+	"elixir": {
+		Command: "mix test",
+	},
+	"elm": {
+		Command: "elm-test",
+	},
+	"emacs-lisp": {
+		Command: "emacs -batch -l ert -l *-test.el -f ert-run-tests-batch-and-exit",
+	},
+	"erlang": {
+		Command: "rebar3 eunit",
+	},
+	"fortran": {
+		Command: "make",
+	},
+	"fsharp": {
+		Command: "dotnet test",
+	},
+	"gleam": {
+		Command: "gleam test",
+	},
+	"go": {
+		Command: "go test",
+	},
+	"groovy": {
+		Command: "gradle test",
+	},
+	"haskell": {
+		Command: "stack test",
+	},
+	"java": {
+		Command: "gradle test",
+	},
+	"javascript": {
+		Command: "npm run test",
+	},
+	"jq": {
+		Command: "bats {{test_files}}",
+	},
+	"julia": {
+		Command: "julia runtests.jl",
+	},
+	"kotlin": {
+		Command:        "./gradlew test",
+		WindowsCommand: "gradlew.bat test",
+	},
+	"lfe": {
+		Command: "make test",
+	},
+	"lua": {
+		Command: "busted",
+	},
+	"mips": {
+		Command: "java -jar /path/to/mars.jar nc runner.mips impl.mips",
+	},
+	"nim": {
+		Command: "nim r {{test_files}}",
+	},
+	// objective-c: tests are run via XCode. There's a CLI option (ruby gem `objc`), but the docs note that this is an inferior experience
+	"ocaml": {
+		Command: "make",
+	},
+	"perl5": {
+		Command: "prove .",
+	},
+	// pharo-smalltalk: tests are run via IDE
+	"php": {
+		Command: "phpunit {{test_files}}",
+	},
+	// plsql: test are run via a "mounted oracle db"
+	"powershell": {
+		Command: "Invoke-Pester",
+	},
+	"prolog": {
+		Command: "swipl -f {{solution_files}} -s {{test_files}} -g run_tests,halt -t 'halt(1)'",
+	},
+	"purescript": {
+		Command: "spago test",
+	},
+	"python": {
+		Command: "python3 -m pytest -o markers=task {{test_files}}",
+	},
+	"r": {
+		Command: "Rscript {{test_files}}",
+	},
+	"racket": {
+		Command: "raco test {{test_files}}",
+	},
+	"raku": {
+		Command: "prove6 {{test_files}}",
+	},
+	"reasonml": {
+		Command: "npm run test",
+	},
+	"red": {
+		Command: "red {{test_files}}",
+	},
+	"ruby": {
+		Command: "ruby {{test_files}}",
+	},
+	"rust": {
+		Command: "cargo test --",
+	},
+	"scala": {
+		Command: "sbt test",
+	},
+	// scheme: docs present 2 equally valid test methods (`make chez` and `make guile`). So I wasn't sure which to pick
+	"sml": {
+		Command: "poly -q --use {{test_files}}",
+	},
+	"swift": {
+		Command: "swift test",
+	},
+	"tcl": {
+		Command: "tclsh {{test_files}}",
+	},
+	"typescript": {
+		Command: "yarn test",
+	},
+	// unison: tests are run from an active UCM session
+	"vbnet": {
+		Command: "dotnet test",
+	},
+	// vimscript: tests are run from inside a vim session
+	"vlang": {
+		Command: "v -stats test run_test.v",
+	},
+	"wasm": {
+		Command: "npm run test",
+	},
+	"wren": {
+		Command: "wrenc {{test_files}}",
+	},
+	"x86-64-assembly": {
+		Command: "make",
+	},
+	"zig": {
+		Command: "zig test {{test_files}}",
+	},
+}
diff --git a/workspace/test_configurations_test.go b/workspace/test_configurations_test.go
new file mode 100644
index 0000000..97ff409
--- /dev/null
+++ b/workspace/test_configurations_test.go
@@ -0,0 +1,103 @@
+package workspace
+
+import (
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestGetCommand(t *testing.T) {
+	testConfig, ok := TestConfigurations["elixir"]
+	assert.True(t, ok, "unexpectedly unable to find elixir test config")
+
+	cmd, err := testConfig.GetTestCommand()
+	assert.NoError(t, err)
+
+	assert.Equal(t, cmd, "mix test")
+}
+
+func TestWindowsCommands(t *testing.T) {
+	testConfig, ok := TestConfigurations["cobol"]
+	assert.True(t, ok, "unexpectedly unable to find cobol test config")
+
+	cmd, err := testConfig.GetTestCommand()
+	assert.NoError(t, err)
+
+	if runtime.GOOS == "windows" {
+		assert.Contains(t, cmd, ".ps1")
+		assert.NotContains(t, cmd, ".sh")
+	} else {
+		assert.Contains(t, cmd, ".sh")
+		assert.NotContains(t, cmd, ".ps1")
+	}
+}
+
+func TestGetCommandMissingConfig(t *testing.T) {
+	testConfig, ok := TestConfigurations["ruby"]
+	assert.True(t, ok, "unexpectedly unable to find ruby test config")
+
+	_, err := testConfig.GetTestCommand()
+	assert.Error(t, err)
+	// any assertions about this error message have to work across all platforms, so be vague
+	// unix: ".exercism/config.json: no such file or directory"
+	// windows: "open .exercism\config.json: The system cannot find the path specified."
+	assert.Contains(t, err.Error(), filepath.Join(".exercism", "config.json:"))
+}
+
+func TestIncludesSolutionAndTestFilesInCommand(t *testing.T) {
+	testConfig, ok := TestConfigurations["prolog"]
+	assert.True(t, ok, "unexpectedly unable to find prolog test config")
+
+	// this creates a config file in the test directory and removes it
+	dir := filepath.Join(".", ".exercism")
+	defer os.RemoveAll(dir)
+	err := os.Mkdir(dir, os.ModePerm)
+	assert.NoError(t, err)
+
+	f, err := os.Create(filepath.Join(dir, "config.json"))
+	assert.NoError(t, err)
+	defer f.Close()
+
+	_, err = f.WriteString(`{ "blurb": "Learn about the basics of Prolog by following a lasagna recipe.", "authors": ["iHiD", "pvcarrera"], "files": { "solution": ["lasagna.pl"], "test": ["lasagna_tests.plt"] } } `)
+	assert.NoError(t, err)
+
+	cmd, err := testConfig.GetTestCommand()
+	assert.NoError(t, err)
+	assert.Equal(t, cmd, "swipl -f lasagna.pl -s lasagna_tests.plt -g run_tests,halt -t 'halt(1)'")
+}
+
+func TestIncludesTestFilesInCommand(t *testing.T) {
+	testConfig, ok := TestConfigurations["ruby"]
+	assert.True(t, ok, "unexpectedly unable to find ruby test config")
+
+	// this creates a config file in the test directory and removes it
+	dir := filepath.Join(".", ".exercism")
+	defer os.RemoveAll(dir)
+	err := os.Mkdir(dir, os.ModePerm)
+	assert.NoError(t, err)
+
+	f, err := os.Create(filepath.Join(dir, "config.json"))
+	assert.NoError(t, err)
+	defer f.Close()
+
+	_, err = f.WriteString(`{ "blurb": "Learn about the basics of Ruby by following a lasagna recipe.", "authors": ["iHiD", "pvcarrera"], "files": { "solution": ["lasagna.rb"], "test": ["lasagna_test.rb", "some_other_file.rb"], "exemplar": [".meta/exemplar.rb"] } } `)
+	assert.NoError(t, err)
+
+	cmd, err := testConfig.GetTestCommand()
+	assert.NoError(t, err)
+	assert.Equal(t, cmd, "ruby lasagna_test.rb some_other_file.rb")
+}
+
+func TestRustHasTrailingDashes(t *testing.T) {
+	testConfig, ok := TestConfigurations["rust"]
+	assert.True(t, ok, "unexpectedly unable to find rust test config")
+
+	cmd, err := testConfig.GetTestCommand()
+	assert.NoError(t, err)
+
+	assert.True(t, strings.HasSuffix(cmd, "--"), "rust's test command should have trailing dashes")
+}