SlowGuess/ACL_2020_OCR huggingface dataset:
{xiye, qchen, isil, gdurrett}@cs.utexas.edu
Existing datasets for regular expression (regex) generation from natural language are limited in complexity; compared to regex tasks that users post on StackOverflow, the regexes in these datasets are simple, and the language used to describe them is not diverse. We introduce STRUCTUREDREGEX, a new regex synthesis dataset differing from prior ones in three aspects. First, to obtain structurally complex and realistic regexes, we generate the regexes using a probabilistic grammar with pre-defined macros observed from real-world StackOverflow posts. Second, to obtain linguistically diverse natural language descriptions, we show crowdworkers abstract depictions of the underlying regex and ask them to describe the pattern they see, rather than having them paraphrase synthetic language. Third, we augment each regex example with a collection of strings that are and are not matched by the ground truth regex, similar to how real users give examples. Our quantitative and qualitative analysis demonstrates the advantages of STRUCTUREDREGEX over prior datasets. Further experimental results using various multimodal synthesis techniques highlight the challenge presented by our dataset, including non-local constraints and multi-modal inputs. $^{1}$
Regular expressions (regexes) are known for their usefulness and wide applicability, and yet they are hard to understand and write, even for many programmers (Friedl, 2006). Recent research has therefore studied how to construct regexes from natural language (NL) descriptions, leading to the emergence of NL-to-regex datasets including
Figure 1: Our dataset collection process. A regex is sampled from our grammar, then we render an abstract figure and generate distinguishing positive/negative examples. We present the figure and examples to crowd-workers to collect natural language descriptions.
KB13 (Kushman and Barzilay, 2013) and NL-TURK (Locascio et al., 2016). However, KB13 is small in size, with only 814 NL-regex pairs with even fewer distinct regexes. Locascio et al. (2016) subsequently employed a generate-and-paraphrase procedure (Wang et al., 2015) to create the larger NL-TURK dataset. However, the regexes in this dataset are very simple, and the descriptions are short, formulaic, and not linguistically diverse because of the paraphrasing annotation procedure (Herzig and Berant, 2019). As a result, even when models achieve credible performance on these datasets, they completely fail when evaluated on the STACKOVERFLOW dataset (Ye et al., 2019), a real-world dataset collected from users seeking help on StackOverflow. The limited size of this dataset (only 62 NL-regex pairs) makes it
$^{1}$ Code and data available at https://www.cs.utexas.edu/~xiye/streg/.
6081
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6081-6094 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(a)	I need to validate the next pattern: starts with “C0” and finish with 4 digits exactly.
and (startwith(<C0>)), endwith(rep(<num>, 4))
(b)	i need regular expression for : one or two digits then "." and one or two digits.
concat (reprange(<num>, 1, 2), concat (<.>, reprange(<num>, 1, 2))
(c)	The input will be in the form a colon (: ) separated tuple of three values. The first value will be an integer, with the other two values being either numeric or a string.
concat (repatleast(<num>, 1), rep (concat (<::>, or (repatleast(<let>, 1),
repatleast(<num>, 1)), 2))
Figure 2: Examples of complex regexes from STACKOVERFLOW. Each regex can be viewed as a set of components composed with a high-level template. Regex (a), for example, can be as viewed the intersection of two constraints specifying the characteristics of the desired regex. (rep means repeat).
unsuitable for large-scale training, and critically, the complexity of regexes it features means that regex synthesis systems must leverage the user-provided positive and negative examples (strings that should be matched or rejected by the target regex) in order to do well.
To enable the development of large-scale neural models in this more realistic regex setting, we present STRUCTUREDREGEX, a new dataset of English language descriptions and positive/negative examples associated with complex regexes. Using a new data collection procedure (Figure 1), our dataset addresses two major limitations in NL-TURK. First, we generate our regexes using a structured probabilistic grammar which includes macro rules defining high-level templates and constructions that involve multiple basic operators. These grammar structures allow us to sample more realistic regexes, with more terminals and operators, while avoiding vacuous regexes. By contrast, the random sampling procedure in NL-TURK leads to simple regexes, and attempting to sample more complex regexes results in atypical regex structures or even contradictory regexes that do not match any string values (Ye et al., 2019). Second, to achieve more realistic language descriptions, we prompt Turkers to write descriptions based on abstract figures that show the desired regexes. We design a set of visual symbols and glyphs to draw a given regex with minimal textual hints. We thereby avoid priming Turkers to a particular way of describing things, hence yielding more linguistically diverse descriptions.
Using this methodology, we collect a total of 3,520 English descriptions, paired with ground truth regexes and associated positive/negative examples. We conduct a comprehensive analysis and demonstrate several linguistic features present in our dataset which do not occur in past datasets. We evaluate a set of baselines, including grammar-
based methods and neural models, on our dataset. In addition, we propose a novel decoding algorithm that integrates constrained decoding using positive/negative examples during inference: this demonstrates the potential of our dataset to enable work at the intersection of NLP and program synthesis. The performance of the best existing approach on STRUCTUREDREGEX only reaches  $37\%$ , which is far behind  $84\%$  on NL-TURK. However, this simple model can nevertheless solve  $13\%$  of the STACKOVERFLOW dataset, indicating that further progress on this dataset can be useful for real-world scenarios.
2 Structured Regex Generation Process
We first describe the structured generative process we adopt to produce the regexes in our dataset. For better readability, we denote regexes using a domain specific language (DSL) similar to regex DSLs in prior work (Locascio et al., 2016; Ye et al., 2019). Our DSL has the same expressiveness as a standard regular language and can be easily mapped back to standard regular expressions.
To collect the NL-TURK dataset, Locascio et al. (2016) sampled regexes using a hand-crafted grammar similar to a standard regex DSL. However, regexes sampled from this process can easily have conflicts (e.g. and (<let>, <num>)) or redundancies (e.g. or (<let>, <low>)). One solution to this problem is rejection sampling, but this still does not yield regexes with compositional, real-world structure.
We show three prominent types of composition observed from STACKOVERFLOW in Figure 2. Each regex above is built by assembling several sub-regexes together according to a high-level template: regex (a) is the intersection of two base regexes expressing constraints, regex (b) is a sequence of three simple parts, and regex (c) is a
2Refer to the appendix for details of our DSL.
starts with "C0" and end with 4 digits
one or two digits then ".” and one or two digits
three delimited values, first will be an integer, with other two being either numeric or a string
Cons	start [end] with (Expr) | not (start [end] with (Expr))	# must (not) start/end with
contain (Expr) | not (contain (Expr))	# must (not) contain
rep( <any>, k) | repatleast( <any>, k) |reprange( <any>, k, k)	# length constraints
AdvStartwithCons | AdvEndwithCons	# adversative macro (e.g., start with capitals except A)
CondContainCons	# conditional macro. (e.g. letter, if contained, must be after a digit)
Comp	Literal | or(Literal, Literal,...)	# literals like digits, letters, strings, or set of literals.
rep (Expr, k) | repatleast (Expr, k) | reprange (Expr, k, k)	# e.g., 3 digits, 2 - 5 letter, etc.
optional (Comp)	# components can be optional.
Figure 3: Examples of our top-level templates and how they cover the three regexes in Figure 2, and overview of sub-regexes (in table) that can possibly be derived from Cons and Comp. Expr as a category here indicates various different constrained sets of sub-regexes. More detail about this structure is available in the full grammar in the appendix.
A string of numbers and digits that must start with a number except "0".
Figure 4: The generation of a deep and complex regex using our grammar. Here, AdvStartwithCons is a macro rule that yields a complex sub-tree with an adversative constraint.
list of three segments delimited by a constant. We observe that these three templates actually capture a wide range of possible regex settings. The first, for example, handles password validation-esque settings where we have a series of constraints to apply to a single string. The second and third reflect matching sequences of fields, which may have shared structured (regex (c)) or be more or less independent (regex (b)).
2.1 Structured Grammar
To generate realistic regexes in these forms, we rely on a structured hand-crafted grammar. The top level of our grammar specifies three templates distilled from STACKOVERFLOW examples: INTERSECTION, CONCATENATION, and SEPARATION, which mimic patterns of real-world regexes. In Figure 3, we show how regexes in Figure 2 can be derived from our templates. The INTERSEC
TION template (left) intersects several base constraints with the and operator; the CONCATENATION template (middle) concatenates several base components with the concat operator. SEPARATION (right) is a more complex type, generating a list of constant-separated INTERSECTION or CONCATENATION regexes which may be identical or share common components.
Across all templates, the components are sub-regexes falling into a few high-level types (notably Cons and Comp), which are depth-limited to control the overall complexity (discussed in Appendix B.2). To make these component regexes more realistic as well, we design several macro rules that expand to more than one operator. The macros are also extracted from real-world examples and capture complex relations like adversative (Figure 4) and conditional (Table 2) relations.
Although our hand-crafted grammar does not cover every possible construction allowed by the regular expression language, it is still highly expressive. Based on manual analysis, our grammar covers  $80\%$  of the real-world regexes in STACK-OVERFLOW, whereas the grammar of NL-TURK only covers  $24\%$  (see Section 4). Note that some constructions apparently omitted by our grammar are equivalent to ones supported by our grammar: e.g., we don't allow a global startwith constraint in the CONCATENATION template, but this constraint can be expressed by having the first component of the concatenation incorporate the desired
6083
constraint.
2.2 Sampling from the Regex Grammar
Although our structural constraints on the grammar already give rise to more realistic regexes, we still want to impose further control over the generative process to mimic properties of real-world regexes. For example, there are sometimes repeating components in CONCATENATION regexes, such as regex (b) from Figure 2.
We encourage such regexes by dynamically modifying the probability of applying the grammar rules while we are expanding a regex based on the status of the entire tree that has currently been induced. For example, suppose we are building regex (b) from Figure 2, and suppose we currently have concat(reprise(), 1,2), concat(<.>, Comp)), where Comp is a non-terminal that needs to be expanded into a sub-regex. Because we already have reprangle(<num>, 1,2) and <.> in the current tree, we increase the probability of expanding Comp to generate these particular two sub-regexes, allowing the model to copy from what it has generated before.
In addition to copying, we also change the sampling distribution when sampling children of certain grammar constructs to control for complexity and encourage sampling of valid regexes. For example, the child of a startwith expression will typically be less complex and compositional than the child of a Comp expression, so we tune the probabilities of sampling compositional AST operators like or appropriately.
3 Dataset Collection
3.1 Positive/Negative Example Generation
The STACKOVERFLOW dataset (Ye et al., 2019) shows that programmers often provide both positive and negative examples to fully convey their intents while specifying a complicated regex. Therefore, we augment our dataset with positive and negative examples for each regex. Our model will use these examples to resolve ambiguity present in the natural language descriptions. However, the examples can also help Turkers to better understand the regexes they are describing during the data collection process.
Figure 5: The process of generating distinguishing negative examples by minorly perturbing each of the sub-regexes in the ground truth regex.
We aim to generate diverse and distinguishing examples similar to human-written ones, which often include corner cases that differentiate the ground truth regex from closely-related spurious ones. We can achieve this by enumerating examples that cover the states in the deterministic finite automaton (DFA) defined by the given regex<sup>4</sup> and reject similar but incorrect regexes. We employ the Automaton Library (Møller, 2017) to generate the examples in our work. Positive examples are generated by stochastically traversing the DFA.
For negative examples, randomly sampling examples from the negation of a given regex will typically produce obviously wrong examples and not distinguishing negative examples as desired. Therefore, we propose an alternative approach shown in Figure 5 for generating negative examples. We apply minor perturbations to the ground truth regex to cause it to accept a set of strings that do not intersect with the set recognized by the original regex. The negative examples can be derived by sampling a positive string from one of these "incorrect" regexes.
For each regex in our dataset, we generate 6 positive examples and 6 negative examples. These numbers are comparable to the average number of examples provided by STACKOVERFLOW users.
3.2 Figure Generation
As stated previously, we avoid the paradigm of asking users to paraphrase machine-generated regex descriptions, as this methodology can yield formulaic and artificial descriptions. Instead, we ask users to describe regexes based on figures that illustrate how the regex is built. We show one example figure of a SEPARATION regex in Figure 6. In general, we abstract a given regex as a series of blocks linked with textual descriptions of its content and constraints. For instance, startwith and endwith are denoted by shading the head or tail of a block. By linking multiple blocks to shared tex
4Recall that although our DSL is tree-structured, it is equivalent in power standard prefixes, and hence our expressions can be mapped to DFAs.
<sup>3</sup>This component reuse bears some similarity to an Adaptor Grammar (Johnson et al., 2007). However, we modify the distributions in a way that violates exchangeability, making it not formally equivalent to one.
6084
Three comma separated segments. The first segment is 2 digits. The other two consist of digits or letters but must start with a letter and contain "0".
Figure 6: An example automatically generated figure of a SEPARATION regex and corresponding description annotated by a Turker.
tual descriptions, we hope to encourage Turkers to notice the correlation and write descriptions accordingly. Finally, we have different textual hints for the same concept: "contain x" in Figure 6 may appear as "have x" elsewhere. These figures are rendered for each regex in the MTurk interface using JavaScript.
3.3 Crowdsourcing
Task We collected the STRUCTUREDREGEX dataset on Amazon Mechanical Turk (MTurk). For each HIT, the Turkers are presented with a regex figure and a set of positive/negative examples. Then, they are asked to write down several sentences describing the regex, as well as one additional positive example that matches the regex. We only accept a description if the submitted positive example is matched by the ground-truth regex; this helps filter out some cases where the Turker may have misunderstood the regex. We show an example HIT in Appendix C.
In early pilot studies, we explored other ways of abstractly explaining Regexes to Turkers, such as providing more examples and an associated set of keywords, yet none of these methods led to users generating sufficiently precise descriptions. By contrast, our figures fully specify the semantics of the regexes while only minimally biasing Turkers towards certain ways of describing them.
We generated 1,200 regexes (400 from each template), assigned each regex to three Turkers, and collected a total of 3,520 descriptions after rejecting HITs. In general, each Turker spent 2 to 3 minutes on each of the HITs, and we set the reward to be  $0.35. The total cost of collecting our dataset was$ 1,512, and the average cost for each description is $0.43.
Dataset	KB13	TURK	STREG	SO
size	824	10000	3520	62
#. unique words	207	557	873	301
avg. NL length	8	12	33	25
avg. reg size	5	5	15	13
avg. reg depth	2.5	2.3	6.0	4.0
Table 1: Statistics of our dataset and prior datasets. Compared to KB13 and NL-TURK, our dataset contains more diverse language and more complex regexes, comparable to the real STACKOVERFLOW dataset.
Quality To ensure the quality of collected responses, we require the Turkers to first take a qualification test which simply requires describing one regex that we have specified in advance. We then check that the description for this regex is sufficiently long and that it contains enough of our manually-written correct base regex concepts.
We manually observed from the responses that various styles were adopted by different Turkers for describing the same type of regexes. For instance, given regex (b) in Figure 2, some Turkers tend to enumerate every component in order, describing it as one or two digits followed by a dot followed by one or two digits; some other Turkers prefer grouping identical components and describing the components out of order, describing it as the first and third parts are one or two digits, and the second part is a dot. These distinct styles lead to a diversity of linguistic phenomena, which is further analyzed in Section 4. Because we aim for high linguistic diversity in our dataset, we prohibited a single Turker from doing more than 300 HITs.
Furthermore, we found anecdotal evidence that the task was engaging for users, which we took as a positive signal for generation quality. We received messages about our HITs from some Turkers telling us that our HIT was "really interesting" and they "enjoyed doing it."
Splitting the Dataset Since our dataset consists of natural language descriptions written by annotators, there is possibly bias introduced by training and testing on the same annotators (Geva et al., 2019). Therefore, in addition to the standard Train/Development/Test splits, we also form a Test-E (excluded) which consists only of annotations from annotators unseen in the training set. We ensure that Train, Dev, and both two test sets (Test and Test-E) have mutually exclusive regexes from each other (Test and Test-E can have common regexes), and Test-E is annotated entirely by
TURK	STREG	Example NL from STREG
multi-sentence ambiguity	0%	70%	The string has 6 or more characters. The string must start with a digit.
2.3%	20.6%	The sequence starts with a letter followed by 2 numbers.
abstraction	0%	13.3%	The first part of a single string consists of 1 or more “0” followed by 2 capital letters. The second part of the string must follow the same rules.
non-local constraint	0%	16.7%	There are 3 dash separated strings. The first is 1 to 4 “A”. The second and third consist of 1 or 2 “x” followed by 1 to 3 numbers and 2 letters.
coreference	5.1%	29.7%	The string starts with a number. It ends with 1 to 4 lower or capital letters.
condition relation	0%	3.5%	If there is a capital letter it must be after a digit.
adversative relation	0%	3.7%	The string starts with capital letter but it should not be a “A”.
a disjoint set of annotators from those who annotated the training or development set. The final size of the splits are: 2173 (61.7%), 351 (10.0%), 629 (17.9%), 367 (10.4%).
4 Dataset Analysis
We demonstrate the advantages of our dataset over prior datasets (Kushman and Barzilay, 2013; Locascio et al., 2016) through both quantitative and qualitative analysis.
We list the key statistics of our dataset as well as KB13 and NL-TURK for comparison in Table 1. Compared to past synthetic datasets, our dataset has more diverse and sophisticated language. The average NL length of our dataset is twice as long as that of NL-TURK, and the descriptions contain many more unique words even though our dataset contains fewer regexes. In addition, our dataset contains more complex regexes that are closer to the complexity of real-world regexes found on StackOverflow, whereas regexes in previous datasets are significantly simpler.
Manual Analysis We further manually analyze 150 descriptions from past synthetic datasets and our dataset. Table 2 lists the proportion of descriptions containing each of several phenomena: examples that are multi-sentence, examples with clear syntactic or semantic ambiguity, examples using abstraction to refer to different parts of the regex, examples invoking non-local constraints, and examples with nontrivial coreference. The language from our dataset is organic and diverse, since we allow Turkers to compose their own descriptions. We find that macros and complex constraints in our structured grammar can successfully trigger interesting language. For instance, the abstraction reflects repetition in concatenation regexes, and the bottom part of Table 2 reflects the
Table 2: Qualitative analysis on 150 descriptions from NL-TURK and our dataset (50 from each template). We show the percentage of examples containing each phenomenon. Our dataset features more of these challenging linguistic phenomena compared to prior synthetic datasets.
KB13	TURK	STREG
Word Coverage	27.1%	34.4%	55.9%
Regex Coverage	23.5%	23.5%	84.3%
Table 3: Distribution mismatch analysis with respect to STACKOVERFLOW on past datasets and our dataset. Our dataset covers significantly more words and regexes, and is closer to the real-world dataset.
complex macros.
Furthermore, the complex and ambiguous language highlights the necessity of including examples together with language to fully specify a regex. For instance, ambiguity is common in our descriptions. However, many of the ambiguous descriptions can be resolved with the help of examples. Concretely, the description for ambiguity from Table 2 can be easily interpreted as startwith(concat(<let>, repeat(<num>, 2)) while the ground truth is concat(<let>, repeat(<num>, 2)). By simply adding one negative example, "a123", the ground truth can be distinguished from the spurious regex.
Comparison to STACKOVERFLOW Since our goal was to produce realistic regex data, we analyze how well the real-world STACKOVERFLOW dataset is covered by data from STRUCTUREDREGEX compared to other datasets (Kushman and Barzilay, 2013; Locascio et al., 2016). We ignore 11 of the STACKOVERFLOW examples that involve the high-level decimal concept, which is beyond the scope of our dataset and past synthetic datasets. In addition, we anonymize all the constants and integer parameters (e.g., repeat  $(<x>,9)$  is anonymized as repeat(const,int)). The statistics (Table 3) suggest that our dataset is more highly similar to real-world regexes on StackOverflow, especially in terms of regex distribution.
6086
We evaluate the accuracy of both existing grammar-based approaches and neural models, as well as a novel method that targets the multimodal nature of our dataset.