Package: turnt
Tiny Unified Runner N' Tester (Turnt)Turnt is a simple snapshot testing tool inspired byCramandLLVM's lit.
It's good for testing things that translate text files to other text files, like compilers.
The idea is that each test is one input file, and you want to run a command and check that it still matches the saved output file.To use it:Configure.Decide what command you want to test.
Make aturnt.tomlconfig file and putcommand = "mycmd {filename}"in it to pass each test file as an argument tomycmd.Create a test.Just write an input file next to yourturnt.toml.
We'll call itfoo.t.Take a snapshot.Runturnt --save foo.tto executemycmd foo.tand save the standard output intofoo.out.
You might want to take a look at this output to make sure it's what you expect.
Then you check both the inputfoo.tand outputfoo.outinto version control.Test your work.Now that you have a test in place, keep working.
Useturnt *.tto run all your tests and confirm that the output still matches.
If there's a mismatch, you can doturnt --diffto see the changes.
(Or if you're confident, tryturnt --savefollowed bygit diff.)Turnt's philosophy is to minimize the effort it takes to write new tests so you can quickly build up lots of them.
You don't write any custom logic to check results; you just record the complete "golden" output for each test.Compared to ordinary unit testing, "snapshot" tests incur the mental effort of manually inspecting diffs when things change.
In return, it's easier to expand test coverage.
Snapshots also act as a crude form of documentation because every test is a complete, valid input to your program.InstallThis is a Python 3 tool.
Install it withpip:$ pip install --user turntOr, if you want to work on Turnt, you can installFlit, clone this repository, and type this to get a "live" installation with a symlink:$ flit install --symlink --userConfiguringTurnt looks for a configuration file calledturnt.tomlin any of the ancestor directories of your test file.
It can be alongside the test file or in any containing directory.
It's aTOMLfile that looks like something this:command = "mycmd {args} < {filename}"
return_code = 42
output.txt = "result.txt"commandSetcommandto a shell command to run on a given test file.
This is the only setting that is truly required.The command is atemplate; Turnt will fill in these values:{filename}: The path to the test file, relative to the working directory.{base}: The basename of the test file, with the extension removed.{args}: Some extra arguments that the test or user provides.
(See theARGS:override and the--argscommand-line option below.)The working directory for the command is the location of theturnt.tomlconfiguration file, if any.
If there's no configuration file, then it's the location of the test file itself.return_codeBy default, Turnt expects the test command to succeed, i.e., exit with status code 0.
Setreturn_codeto a different status if you expect failure.outputBy default, Turnt captures the standard output stream from your test command.
If your command produces other output files "on the side" or you want to capture the standard error stream, you can configure theoutputtable.outputis a mapping fromsnapshot extensionstocollected filenames.
For example, this TOML configuration:output.txt = "result.txt"means that running the command will produce a file calledresult.txt, and we want to save that file in a snapshot called<test-name>.txt.In place of a filename, use-to indicate the command's standard output and2to indicate its standard error.
The default behaves like this configuration:output.out = "-"which captures stdout and saves it in<test-name>.out.
Definingoutputinturnt.tomldisables this default behavior; you can include it explicitly if you want it alongside other outputs.binaryBy default, Turnt looks inside test files for overrides (see below).
This won't work if your test inputs are binary (non-text) files (Turnt will warn you and proceed with no overrides).
Setbinary = trueto suppress this search for overrides altogether.out_dirOptionally put expected test outputs in a different directory.
By default, this is., i.e., expected output files go "next to" test files.
Set this to a different directory, relative to the test file, to keep them somewhere else.Per-Test OverridesSometimes you need to alter the setup for a specific test file.
Turnt looks for some overrides embedded in the test file itself: for example, you might put them in a comment at the top of a test program.Put these things into your test file to override the configuration:CMD: <command>overridescommandfrom the configuration.ARGS: <arguments>adds arguments to a configured command.
Turnt puts this string in where the command uses{args}.OUT: <ext> <filename>overridesoutputfrom the configuration.
You can specify multiple files this way: one line per file.RETURN: <code>overrides the expected exit status.Multiple EnvironmentsTurnt is mostly about running one command on many input files.
Sometimes, however, you need to run several commands on each file.
This can be especially useful fordifferential testing:when you want to check that multiple things behave the same way by checking that they produce the same input when you give them the same input.You can create multiple environments in your configuration file under theenvstable.
The table maps environmentnamesto sets of configuration options that are just like thetop-level configuration described above.
For example:[envs.baseline]
command = "interp -g {filename}"

[envs.optimized]
command = "compile -O3 {filename} ; ./a.out"Each environment can have the full complement of configuration options described above:
for example,command,output, andreturn_code.
When you runturnton some files, it will run all the environments on all the files.
There is also a Booleandefaultflag to turn off an environment by default; that way, you can only use it by asking for it with the-eflag (see below).This example is a differential testing setup because both environments share the same (default) snapshot file:
a testfoo.twill look for its stdout snapshot infoo.out.
If the two environments don't match, at least one will fail.
If you want a more standard (non-differential) setup, just set theoutputconfiguration differently for the two environments, like this:[envs.interp]
command = "interp -g {filename}"
output.res = "-"

[envs.profile]
command = "profile {filename}"
output.prof = "-"Directory TestsA Turnt test is usually just a single input file, but you can also organize multiple related files into a directory.
Use the directory the same way as you would a single file:
pass its path to theturntcommand, and the path will appear as the{filename}for the configured command.
So you might configure your test command like this:command = "mycmd {filename}/test.c"if you want each test directory to contain a file calledtest.c.Turnt puts snapshotsinsidethe test directory instead of adjacent to it.
It names themout.<extension>in that directory.There are two configuration options just for dealing with directory tests:out_base.
The basename for output files in directory tests: by default,out.opts_file.
The filename to read inside of a directory test to search for embedded overrides.
In our above example, you could set this totest.cto look forARGS:and friends in that file, oropts.txtto look for them in a separate file on the side.Command-Line InterfaceThe most commonturntcommand-line options you'll need while running and updating tests are:--save: Bless the current output from each test as the "correct" output, saving it to the output file that you'll want to check into version control.--diff: Show diffs between the actual and expected output for each test.
(The diff goes to stderr while theTAPresults remain on stdout.)You also might enjoy:--parallelor-j: Run your tests faster using parallel threads.These options are useful when working with one specific test file:--verboseor-v: Disable Turnt's default behavior where it will suppress test commands' stderr output. The result is more helpful but harder to read.--printor-p: Instead of checking test results, just run the command and show the output directly. This can be useful (especially in combination with-v) when iterating on a test interactively.--argsor-a: Override the{args}string in the test command.These options lets you switch between different test environments:--envor-e: Give the name of a configuredenvironmentto run. Use this multiple times to run multiple environments. By default, Turnt runs all the configured environments for every test. Use the special namedefaultfor the top-level, unnamed test environment.--configor-c: Look for this config filename instead of the defaultturnt.toml.TAPTurnt outputs results in the machine-readableTAPformat.
To make the output more pleasant to read, you can pipe it into a tool liketap-difflet,tap-dot, orfaucet:$ npm install -g tap-difflet
$ turnt *.t | tap-diffletCreditsTurnt is byAdrian SampsonandAlexa VanHattum.
We made it to test various research compilers inCapra.
The license isMIT.