Package: grammarinator
ANTLRv4 grammar-based test generatorGrammarinatoris a random test generator / fuzzer that creates test cases
according to an inputANTLRv4 grammar. The motivation behind this
grammar-based approach is to leverage the large variety of publicly
availableANTLR v4 grammars.Thetrophy pageof the found issues is available from the wiki.RequirementsPython>= 3.7JavaSE >= 7 JRE or JDK (the latter is optional)InstallTo useGrammarinatorin another project, it can be added tosetup.cfgas
an install requirement (if usingsetuptoolswith declarative config):[options]install_requires=grammarinatorTo installGrammarinatormanually, e.g., into a virtual environment, usepip:pip install grammarinatorThe above approaches install the latest release ofGrammarinatorfromPyPI.
Alternatively, for the development version, clone the project and perform a
local install:pip install .UsageAs a first step,Grammarinatortakes anANTLR v4 grammarand creates a test
generator script in Python3. Grammarinator supports a subset of the features
of the ANTLR grammar which is introduced in the Grammar overview section of the
documentation. The produced generator can be subclassed later to customize it
further if needed.Basic command-line syntax of test generator creation:grammarinator-process <grammar-file(s)> -o <output-directory> --no-actionsNotesGrammarinatoruses theANTLR v4 grammarformat as its input, which
makes existing grammars (lexer and parser rules) easily reusable. However,
because of the inherently different goals of a fuzzer and a parser, inlined
code (actions and conditions, header and members blocks) are most probably
not reusable, or even preventing proper execution. For first experiments
with existing grammar files,grammarinator-processsupports the
command-line option--no-actions, which skips all such code blocks
during fuzzer generation. Once inlined code is tuned for fuzzing, that
option may be omitted.After having generated and optionally customized a fuzzer, it can be executed
by thegrammarinator-generatescript (or by manually instantiating it in a
custom-written driver, of course).Basic command-line syntax ofgrammarinator-generate:grammarinator-generate <generator> -r <start-rule> -d <max-depth> \
  -o <output-pattern> -n <number-of-tests> \
  -t <transformer1> -t <transformer2>Beside generating test cases from scratch based on the ANTLR grammar,
Grammarinator is also able to recombine existing inputs or mutate only a small
portion of them. To use these additional generation approaches, a population of
selected test cases has to be prepared. The preparation happens with thegrammarinator-parsetool, which processes the input files with an ANTLR
grammar (possibly with the same one as the generator grammar) and builds
grammarinator tree representations from them (with .grt extension). Having a
population of such .grt files,grammarinator-generatecan make use of them
with the--populationcli option. If the--populationoption is set,
then Grammarinator will choose a strategy (generation, mutation, or
recombination) randomly at the creation of every new test case. If any of the
strategies is unwanted, they can be disabled with the--no-generate,--no-mutateor--no-recombineoptions.Basic command line syntax ofgrammarinator-parse:grammarinator-parse <grammar-file(s)> -r <start-rule> \
  -i <input_file(s)> -o <output-directory>NotesReal-life grammars often use recursive rules to express certain patterns.
However, when using such rule(s) for generation, we can easily end up in an
unexpectedly deep call stack. With the--max-depthor-doptions,
this depth - and also the size of the generated test cases - can be
controlled.Another specialty of the ANTLR grammars is that they support so-called
hidden tokens. These rules typically describe such elements of the target
language that can be placed basically anywhere without breaking the syntax.
The most common examples are comments or whitespaces. However, when using
these grammars - which don’t define explicitly where whitespace may or may
not appear in rules - to generate test cases, we have to insert the missing
spaces manually. This can be done by applying a serializer (with the-soption) to the tree representation of the output tests. A simple serializer
- that inserts a space after every unparser rule - is provided byGrammarinator(grammarinator.runtime.simple_space_serializer).In some cases, we may want to postprocess the output tree itself (without
serializing it). For example, to enforce some logic that cannot be expressed
by a context-free grammar. For this purpose the transformer mechanism can be
used (with the-toption). Similarly to the serializers, it will take a
tree as input, but instead of creating a string representation, it is
expected to return the modified (transformed) tree object.As a final thought, one must not forget that the original purpose of
grammars is the syntax-wise validation of various inputs. As a consequence,
these grammars encode syntactic expectations only and not semantic rules. If
we still want to add semantic knowledge into the generated test, then we can
inherit custom fuzzers from the generated ones and redefine methods
corresponding to lexer or parser rules in ways that encode the required
knowledge (e.g.:HTMLCustomGenerator).Working ExampleThe repository contains a minimalexampleto generate HTML files. To give it
a try, run the processor first:grammarinator-process examples/grammars/HTMLLexer.g4 examples/grammars/HTMLParser.g4 \
  -o examples/fuzzer/Then, use the generator to produce test cases:grammarinator-generate HTMLCustomGenerator.HTMLCustomGenerator -r htmlDocument -d 20 \
  -o examples/tests/test_%d.html -n 100 \
  -s HTMLGenerator.html_space_serializer \
  --sys-path examples/fuzzer/CompatibilityGrammarinatorwas tested on:Linux (Ubuntu 16.04 / 18.04 / 20.04)OS X / macOS (10.12 / 10.13 / 10.14 / 10.15 / 11)Windows (Server 2012 R2 / Server version 1809 / Windows 10)CitationsBackground onGrammarinatoris published in:Renata Hodovan, Akos Kiss, and Tibor Gyimothy. Grammarinator: A Grammar-Based
Open Source Fuzzer.
In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating
Test Case Design, Selection, and Evaluation (A-TEST 2018), pages 45-48, Lake
Buena Vista, Florida, USA, November 2018. ACM.https://doi.org/10.1145/3278186.3278193Copyright and LicensingLicensed under the BSD 3-ClauseLicense.