SlowGuess/ACL_2020_OCR huggingface dataset:
Existing Approaches SEMANTIC-UNIFY (Kushman and Barzilay, 2013) is a grammar-based approach that relies on a probabilistic combinatory categorical grammar to build the regexes. DEEPREGEX (Locascio et al., 2016) directly translates natural language descriptions into regexes using a seq-to-seq model enhanced with attention (Luong et al., 2015) without considering examples. We re-implemented DEEPREGEX with slightly different hyperparameters; we refer to our re-implementation as DEEPREGEX (OURS). DEEPREGEX+FILTER (Ye et al., 2019) adapts DEEPREGEX so as to take examples into account by simply filtering the  $k$ -best regexes based on whether a regex accepts all the positive examples and rejects all the negative ones.
Example-Guided Decoding Although DEEP-REGEX+FILTER is able to take advantage of positive and negative string examples, these examples are completely isolated in the training and inference phase. We propose to make use of examples during inference with the technique of over- and under- approximation (Lee et al., 2016) used in the program synthesis domain. The core idea of our approach is that, for each partially completed regex during decoding, we use the approximation technique to infer whether the regex can possibly match all positive or reject all negative examples. If this is impossible, we can prune this partial regex from our search. This approach allows us to more effectively explore the set of plausible regexes without increasing the computational budget or beam size.
As an example, consider the ground truth regex and (startwith(<low>), endwith(<num>) with one corresponding positive example "00x". Suppose that the decoder has so far generated the incomplete regex and (startwith(<cap>), . To produce a syntactically valid regex, the decoder needs to generate a second argument for the and. By appending star (<any>) as its second argument, we can see that there is no completion here that will accept the given positive example, allowing us to reject this regex from the beam. Under-approximation works analogously,
Approach	KB13	TURK	STREG
SEMANTIC-UNIFY	65.5%	38.6%	1.8%
DEEPEREGEX (Locascio et al.)	65.6%	58.2%	-
DEEPEREGEX (Ours)	66.5%	60.2%	24.5%
DEEPEREGEX + FILTER	77.7%	83.8%	37.2%
Table 4: DFA-equivalent accuracy on prior datasets and our dataset. The performance on our dataset using any model is much lower than the performance on existing datasets.
completing Regexes with maximally restrictive arguments and checking that negative examples are rejected.
We integrate the aforementioned technique in the beam decoding process by simply pruning out bad partial derivations at each timestep. We refer to this approach as DEEPREGEX + APPROX.
6.1 Comparison to Prior Datasets
We evaluate the baseline models on KB13, NL-TURK, and our dataset (Table 4). The results show that our dataset is far more challenging compared to existing datasets. Traditional grammar baseline can scarcely solve our dataset. The best baseline, DEEPREGEX + FILTER, achieves more than  $77.7\%$  on KB13 and  $83.8\%$  NL-TURK when these datasets are augmented with examples, but can only tackle  $37.2\%$  of our dataset. Additionally, the comparison between DEEPREGEX and DEEP-REGEX + FILTER demonstrates that simply filtering the outputs of neural model leads to a substantial performance boost on all the datasets. This supports the effectiveness of the way we specify regexes, i.e., using both natural language descriptions and examples.
6.2 Detailed Results on STRUCTUREDREGEX
Table 5 shows the detailed accuracy regarding different regex templates on both Test and Test-E sets. Our DEEPREGEX + APPROX achieves best accuracy with  $5.6\%$  and  $7.9\%$  improvement over DEEPREGEX + FILTER on Test and Test-E, respectively, since it can leverage examples more effectively using over- and under- approximations during search.
Accuracy varies on different types of regexes. Generally, models perform the best on concatenation regexes, slightly worse on intersection regexes, and the worst on separation regexes. Concatenation regexes usually have straightforward
6087
Approach	Test	Test-E
Agg	Int	Cat	Sep	Agg	Int	Cat	Sep
SEMANTIC-UNIFY	2.1%	2.9%	3.1%	0.0%	1.4%	1.6%	2.4%	0.0%
DEEPREGEX (Ours)	27.8%	20.7%	42.2%	19.2%	18.8%	18.0%	23.6%	14.8%
DEEPREGEX + FILTER	42.6%	38.9%	55.2%	32.3%	28.1%	32.0%	32.5%	19.7%
DEEPREGEX + APPROX	48.2%	45.7%	59.6%	37.9%	36.0%	39.3%	40.7%	27.9%
Table 5: Results for models trained and tested on STRUCTUREDREGEX. Using the examples (the latter two methods) gives a substantial accuracy boost, and DEEPREGEX + APPROX is better than the post-hoc FILTER method, but still only achieves  $48.2\%$  accuracy on Test and  $36.0\%$  on Test-E. Separation regexes are more difficult than the other two classes, and performance for all models drops on Test-E.
Train Set	Model DEEPREGEX	Acc	Equiv Found	Consistent Found
TURK	w/o Example	0.0%	0.0%	7.8%
STREG	+ FILTER	9.8%	9.8%	21.6%
STREG	+APPROX	13.7%	17.6%	37.7%
Table 6: The performance on STACKOVERFLOW-51 with models trained on NL-TURK and our dataset. We report the fraction of examples where a DFA-equivalent regex is found (Acc), where a DFA-equivalent regex is found in the  $k$ -best list, and where a regex consistent with the examples appears in the  $k$ -best list. Models trained on NL-TURK do not perform well in this setting, while our models can solve some examples.
descriptions in the form of listing simple components one by one. Intersection descriptions can be more complicated because of the high-level macros specified by our grammar. Separation descriptions are the most complex ones that often involve coreferences and non-local features. Performance on Test-E is  $12\%$  lower than on Test for the models haven't been trained on patterns of the unseen annotators.
6.3 Transferability Results
Finally, we investigate whether a model trained on our dataset can transfer to the STACKOVERFLOW dataset. As in Section 4, we ignore instances requiring the decimal concept and only evaluate on the subset of STACKOVERFLOW with 51 instances. We compare our dataset with NL-TURK for this task. As shown in Table 6, DEEP-REGEX trained on NL-TURK completely fails on STACKOVERFLOW and even fails to predict reasonable regexes that are consistent with the examples. This is caused by the fact that the NL-TURK dataset contains formulaic descriptions and shallow regexes that are not representative of real-world tasks. DEEP-REGEX trained on our dataset can at least achieve  $9.8\%$  accuracy on STACKOVERFLOW dataset because the English descrip
tions in this dataset better match the desired task. Our DEEPREGEX + APPROX model successfully solves  $13.7\%$  and finds consistent regexes for  $38\%$  of the tasks, which is credible given that the performance of the same model on Test-E set is only  $30\%$ . Some additional challenges in STACKOVERFLOW are instances involving large numbers of constants or slightly more formal language since the SO users are mainly programmers. However, we believe the transfer results here show that improved performance on our dataset may transfer to STACKOVERFLOW as well, since some of the challenges also present in our Test-E set (e.g., unseen language).
6.4 Human Performance Estimate
It is difficult to hire Turkers to estimate a human performance upper bound, because our task requires reckoning with both the descriptions and positive/negative examples. Unlike many NLP tasks where an example with ambiguous language is fundamentally impossible, here the examples may actually still allow a human to determine the correct answer with enough sleuthing. But to perform this task, crowdworkers would minimally need to be trained to understand the DSL constructs and how they compose, which would require an extensive tutorial and qualification test. To do the task well, Turkers would need a tool to do on-the-fly execution of their proposed regexes on the provided examples.
We instead opted for a lighter-weight verification approach to estimate human performance. We adopted a post-editing approach on failure cases from our model, where we compared the model's output with the input description and examples and corrected inconsistencies.
Specifically, we sample 100 failure examples from the test set (Test plus Test-E) and manually assess the failure cases. We find  $78\%$  of failure cases contain descriptions that describe all com
6088
ponents of the target regexes, but our seq-to-seq models are insufficient to capture these. There are truly some mis- or under-specified examples, such as not mentioning the optionality of one component or mistaking "I" for "l" in constants. An additional  $9\%$  (out of 100) of the errors could be fixed using the provided examples. This leaves roughly  $13\%$  of failure cases that are challenging to solve.
Considering that the model already achieves  $43.6\%$  accuracy on the test set, we estimate human performance is around  $90\%$ .
Data collection in semantic parsing Collecting large-scale data for semantic parsing and related tasks is a long-standing challenge (Berant et al., 2013; Wang et al., 2015). Wang et al. (2015) proposed the generate-and-paraphrase framework, which has been adopted to collect datasets in various domains (Locascio et al., 2016; Ravichander et al., 2017; Johnson et al., 2017). However, this process often biases annotators towards using formulaic language (Ravichander et al., 2017; Herzig and Berant, 2019).
Similar to our work, past work has sought to elicit linguistically diverse data using visual elements for semantic parsing (Long et al., 2016), natural language generation (Novikova et al., 2016), and visual reasoning (Suhr et al., 2017, 2019). However, for these other tasks, the images used are depictions of an inherently graphical underlying world state; e.g., the NLVR dataset (Suhr et al., 2017) and NLVR2 (Suhr et al., 2019) are based on reasoning over the presented images, and the Tangrams dataset (Long et al., 2016) involves describing shape transformations. By contrast, regexes are typically represented as source code; there is no standard graphical schema for depicting the patterns they recognize. This changes the properties of the generated descriptions, leading to higher levels of compositionality and ambiguity because what's being described is not naturally an image.
Program and regex synthesis Recent research has tackled the problem of program synthesis from examples (Gulwani, 2011; Gulwani and Jain,
2017; Alur et al., 2013; Wang et al., 2016; Feng et al., 2018; Devlin et al., 2017; Nye et al., 2019). A closer line of work to ours uses both examples and natural language input (Yaghmazadeh et al., 2017; Ye et al., 2019; Andreas et al., 2018), which involves fundamentally different techniques. However, our work does not rely on the same sort of program synthesizer to build final outputs (Yaghmazadeh et al., 2017; Ye et al., 2019). Moreover, Andreas et al. (2018) only use language at train time, whereas we use NL at both train and test time.
Finally, while several datasets on regex synthesis specifically have been released (Kushman and Barzilay, 2013; Locascio et al., 2016), we are the first to incorporate examples in the dataset. Other methods have been proposed to parse natural language into regex via rule-based (Ranta, 1998), grammar-based (Kushman and Barzilay, 2013), or neural models (Locascio et al., 2016; Zhong et al., 2018; Ye et al., 2019). Notably, Zhong et al. (2018) also generate distinguishing examples to facilitate translation, but they require a trained model to generate examples, and we organically derive examples from the structure of regexes without additional input.
We introduce STRUCTUREDREGEX, a new dataset for regex synthesis from natural language and examples. Our dataset contains compositionally structured regexes paired with linguistically diverse language, and organically includes distinguishing examples. Better methods are needed to solve this dataset; we show that such methods might generalize well to real-world settings.
This work was partially supported by NSF Grant IIS-1814522, NSF Grant SHF-1762299, a gift from Arm, and an equipment grant from NVIDIA. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources used to conduct this research. Thanks as well to the anonymous reviewers for their helpful comments.
Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo MK Martin, Mukund Raghothaman, Sanjit A
In addition, the first author manually wrote regexes for 100 randomly sampled examples and achieved an accuracy of  $95\%$  (higher than the estimate). However, the author also has a strong prior over what synthetic regexes are likely to be in the data.
6089
Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-guided Synthesis. In 2013 Formal Methods in Computer-Aided Design (FMCAD).
Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with Latent Language. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural Program Learning under Noisy I/O. In Proceedings of the International Conference on Machine Learning (ICML).
Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. 2018. Program Synthesis Using Conflict-driven Learning. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI).
Jeffrey EF Friedl. 2006. Mastering Regular Expressions." O'Reilly Media, Inc".
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Sumit Gulwani. 2011. Automating String Processing in Spreadsheets Using Input-output Examples. In Proceedings of the ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL).
Sumit Gulwani and Prateek Jain. 2017. Programming by Examples: PL Meets ML. In Proceedings of the Asian Symposium on Programming Languages and Systems (APLAS).
Jonathan Herzig and Jonathan Berant. 2019. Don't paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A Diagnostic Dataset for Compositional Language and Elementary Visual
Reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. 2007. Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS).
Nate Kushman and Regina Barzilay. 2013. Using Semantic Unification to Generate Regular Expressions from Natural Language. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NACCL).
Mina Lee, Sunbeom So, and Hakjoo Oh. 2016. Synthesizing Regular Expressions from Examples for Introductory Automata Assignments. In Proceedings of the ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE).
Nicholas Locascio, Karthik Narasimhan, Eduardo DeLeon, Nate Kushman, and Regina Barzilay. 2016. Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler Context-Dependent Logical Forms via Model Projections. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Anders Møller. 2017. dk.bricsautomation - finite-state automata and regular expressions for Java. http://www.brics.dk/automaton/.
Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG Data: Pictures Elicit Better Data. In Proceedings of the International Natural Language Generation conference (INLG).
Maxwell Nye, Luke Hewitt, Joshua Tenenbaum, and Armando Solar-Lezama. 2019. Learning to Infer Program Sketches. In Proceedings of the International Conference on Machine Learning (ICML).
Aarne Ranta. 1998. A Multilingual Natural-Language Interface to Regular Expressions. In Finite State Methods in Natural Language Processing.
Abhilasha Ravichander, Thomas Manzini, Matthias Grabmair, Graham Neubig, Jonathan Francis, and Eric Nyberg. 2017. How Would You Say It? Eliciting Lexically Diverse Dialogue for Supervised Semantic Parsing. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL).
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A Corpus of Natural Language for Visual Reasoning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A Corpus for Reasoning about Natural Language Grounded in Photographs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Xinyu Wang, Sumit Gulwani, and Rishabh Singh. 2016. FIDEX: Filtering Spreadsheet Data Using Examples. In Proceedings of the ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOP-SLA).
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a Semantic Parser Overnight. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: Query Synthesis from Natural Language. In Proceedings of the ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA).
Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, and Greg Durrett. 2019. Sketch-Driven Regular Expression Generation from Natural Language and Examples. In arXiv preprint arXiv:1908.05848.
Zexuan Zhong, Jiaqi Guo, Wei Yang, Tao Xie, JianGuang Lou, Ting Liu, and Dongmei Zhang. 2018. Generating regular expressions from natural language specifications: Are we there yet? In the Statistical Modeling of Natural Software Corpora Workshop at the AAAI Conference on Artificial Intelligence (AAAI Workshop).
6091
A Regex DSL
Nonterminals r :=
startwith(r)	r.*
endwith(r)	.*r
contain(r)	.*r.*
not(r)	~r
optional(r)	r?
star(r)	r*
concat(r1, r2)	r1r2
and(r1, r2)	r1&r2
or(r1, r2)	r1|r2
rep(r, k)	r{k}
repatleast(r, k)	r{k,}
reprange(r, k1, k2)	r{k1, k2}
Terminals t :=
<let>	[A-Za-z]
<cap>	[A-Z]
<low>	[a-z]
<num>	[0-9]
<any>	.
<spec>	[-,; .+: !@#_$%&^=^]
<null>	∅
Table 7: Our regex DSL and the corresponding constructions in standard regular language. Our regex DSL is as expressive as and can be easily translated to standard regex syntax.
B Details of Structured Grammar
B.1 Grammar Rules
See Figure 7.
Intersection While building INTERSECTION regexes, we impose context-dependent constraints mainly to avoid combinations of regexes that are redundant or in conflict. Conflicts often occur between a ComposedBy constraint and the other constraints. A ComposedBy constraint indicates the allowed characters; e.g., repeatatleast(or(<let>,<spec>),1) means there can only be letters and special characters in the matched string. Therefore, when we already have such a constraint in the tree, we only allow the terminals to be selected from the valid subset of <let> and <spec> while expanding the other subtrees.
This greatly reduce the chances of yielding empty regexes as well as redundant regexes (e.g., in and(repeatatleast(or(<let>,<spec>), 1), not (contain (<num>))), the second constraint is actually redundant).
Concatenation CONCATENATIONregexes are a sequence of simple components. As stated above,
our grammar encourages the phenomenon of repetition that commonly occurs in real Regexes by copying existing sub-trees.
Separation SEPARATION regexes have several subfields, which can be specified by either INTERSECTION regexes or CONCATENATION regexes, and which are delimited by a constant. The fields of real regexes are often related, i.e., they share common components. For instance, the format of U.S. phone numbers is "xxx-xxx-xxxx" where "x" is a digit. Here the three fields are all digits but differ in length. Similar to the CONCATENATION template, we alter the distribution so as to copy the already generated subtrees.
We also allow a class of SEPARATION with an arbitrary number of identical fields separated by a constant (e.g., a list of comma-separated numbers).
Complexity Control We aim to create a collection of complicated regexes, but we do not wish to make them needlessly complex along unrealistic axes. We assess the complexity of generated regexes using a measure we call semantic complexity, which roughly measures how many factors would need to be specified by a user. Generally, each constraint or components counts for one degree of semantic complexity, e.g., not (contain(x)) and repeat(x,4) are of complexity level one. High-level macro constraints are of complexity level two since they need more verbal explanation. We limit the complexity degrees all of our generated regexes to be strictly no more than six. More details about the number of nodes and depth of our regexes can be found in Section 4.
C HIT Example
Figure 7: Grammar rules for generating regexes in our dataset. Our grammar contains much more rules than a standard regex grammar, and is highly structured in that we have high-level templates and macros.
6093
In this task, you will be writing down descriptions of the patterns you see in a group of strings. For each HIT, you'll be given a figure visually specifying a pattern and a few examples of strings following or not following the pattern to help you to understand it. Please write a description (generally 1-4 sentences) that describes the pattern. In addition, please write one additional string that follows the pattern.
Things to keep in mind:
- Please describe the pattern underlying the string examples, not the sequence of strings itself. Do not write things like "the first line ..., the second line ..."
- Try to be precise about describing the pattern, but also concise. Don't describe the same property of the strings in multiple ways.
- You are not required to use the keywords in the figure. If you can think of another way to express the intent, that's okay.
- Please try to write natural and fluent sentences.
- Additional string example must be different.
Figure 8: HIT prompt for the description writing task. We particularly emphasize in the instructions that Turkers should use precise and original language.
Example strings that follow the pattern:
a51,B457
a74,B23
a09,849
Example strings that do not follow the pattern:
b55,B193
a7,B23
a09,1
6094