SlowGuess/ACL_2020_OCR huggingface dataset:
Programming in Natural Language with  $fuse$ : Synthesizing Methods from Spoken Utterances Using Deep Natural Language Understanding
Sebastian Weigelt, Vanessa Steurer, Tobias Hey, and Walter F. Tichy  
Karlsruhe Institute of Technology  
Institute for Program Structures and Data Organization  
Karlsruhe, Germany  
weigelt@kit.edu, vanessa.steurer@web.de, hey@kit.edu, tichy@kit.edu
The key to effortless end-user programming is natural language. We examine how to teach intelligent systems new functions, expressed in natural language. As a first step, we collected 3168 samples of teaching efforts in plain English. Then we built  $fuse$ , a novel system that translates English function descriptions into code. Our approach is three-tiered and each task is evaluated separately. We first classify whether an intent to teach new functionality is present in the utterance (accuracy:  $97.7\%$  using BERT). Then we analyze the linguistic structure and construct a semantic model (accuracy:  $97.6\%$  using a BiLSTM). Finally, we synthesize the signature of the method, map the intermediate steps (instructions in the method body) to API calls and inject control structures ( $F_1$ :  $67.0\%$  with information retrieval and knowledge-based methods). In an end-to-end evaluation on an unseen dataset  $fuse$  synthesized  $84.6\%$  of the method signatures and  $79.2\%$  of the API calls correctly.
Intelligent systems became rather smart lately. One easily arranges appointments by talking to a virtual assistant or controls a smart home through a conversational interface. Instructing a humanoid robot in this way no longer seems to be futuristic. For the time being, users can only access built-in functionality. However, they will soon expect to add new functionality themselves. For humans, the most natural way to communicate is by natural language. Thus, future intelligent systems must be programmable in everyday language.
Today's systems that claim to offer programming in natural language enable laypersons to issue single commands or construct short scripts (e.g. Mihalcea et al. (2006); Rabinovich et al. (2017)); usually no new functionality is learned. Only a
few addressed learning new functionality from natural language instructions (e.g. Le et al. (2013); Markievicz et al. (2017)). However, even recent approaches still either restrict the language or are (over-)fitted to a certain domain or application.
We propose to apply deep natural language understanding to the task of synthesizing methods from spoken utterances. Our approach combines modern machine learning techniques with information retrieval and knowledge-based methods to grasp the user's intent. As a first step, we have performed a user study to investigate how laypersons teach new functionality with nothing but natural language. In a second step, we develop  $fuse$  (Function Synthesis Executor).  $fuse$  translates teaching efforts into code. On the basis of the gathered data we constructed a three-tiered approach. We first determine, whether an utterance comprises an explicitly stated intent to teach a new skill. Then, we decompose these teaching efforts into distinct semantic parts. We synthesize methods by transferring these semantic parts into a model that represents the structure of method definitions. Finally, we construct signatures, map instructions of the body to API calls, and inject control structures.
The objective of programming in natural language was approached from different perspectives over the years. Quite a few approaches are natural language interfaces to code editors (Price et al., 2000; Begel, 2004; Begel and Graham, 2005; Désilets et al., 2006). However, they assume that users literally dictate source code. Thus, these approaches are intended for developers rather than laypersons. Other approaches such as Voxelurn by Wang et al. (2017) aim to naturalize programming languages to lower the hurdle for programming novices.
Approaches for end-user programming in natu
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4280-4295 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Schematic overview of  $f u_{\mathrm{SE}}$ 's three-tiered approach.
ral language take up the challenge of bridging the semantic gap between informal spoken or written descriptions in everyday language and formal programming languages. Early systems were syntax-based (Winograd, 1972; Ballard and Biermann, 1979; Biermann and Ballard, 1980; Biermann et al., 1983; Liu and Lieberman, 2005). Some were already capable to synthesize short scripts including control structures and comments, e.g. NLP for NLP by Mihalcea et al. (2006). Others take the user in the loop and create scripts with a dialog-driven approach (Le et al., 2013). In further developments intelligent assistants offer their service to assist with programming (Azaria et al., 2016). Often these assistants support multi-modal input, e.g. voice and gestures (Campagna et al., 2017, 2019). Others combine programming in natural language with other forms of end-user programming, such as programming by example (Manshadi et al., 2013) or programming by demonstration (Li et al., 2018).
Some authors such as Landhauer et al. (2017) and Atzeni and Atzori (2018a,b) take a knowledge-based approach by integrating domain and environmental information in the form of ontologies.
Suhr and Artzi (2018) employ a neural network to learn a situational context model that integrates the system environment and the human-system-interaction, i.e. the dialog. Many recent approaches integrate semantic parsing in the transformation process (Guu et al., 2017; Rabinovich et al., 2017; Chen et al., 2018; Dong and Lapata, 2018). Even though the natural language understanding capabilities are often impressive, the synthesized scripts are still (semantically) erroneous in most cases. Additionally, learning of new functionality is not covered by approaches of that category so far.
Programming in natural language is of particular interest in the domain of humanoid robotics (Lauria et al., 2001, 2002; She et al., 2014; Mei et al., 2016). People expect to teach them as they teach human co-workers. Therefore, some authors, e.g.
Markievicz et al. (2017), use task descriptions that were intended to instruct humans to benchmark their approach. However, often the assumed vocabulary is rather technical (Lincoln and Veres, 2012). Thus, the usability for laypersons is limited.
The goal of our work is to provide a system for programming in (spoken) natural language. Laypersons shall be enabled to create new functionality in terms of method definitions by using natural language only. We offer a general approach, i.e. we do not restrict the natural language regarding wording and length. Since spontaneous language often comprises grammatical flaws, disfluencies, and alike, our work must be resilient to these issues.
We decompose the task in three consecutive steps. The rationale behind this decision is as follows. On the one hand, we can implement more focused (and precise) approaches for each task, e.g. using machine learning for one and information retrieval for another. On the other hand, we are able to evaluate and optimize each approach individually. The stages of our three-tiered approach are the following (see Figure 1 for an example):
1. Classification of teaching efforts: Determine whether an utterance comprises an explicitly stated teaching intent or not.
2. Classification of the semantic structure: Analyze (and label) the semantic parts of a teaching sequence. Teaching sequences are composed of a declarative and a specifying part as well as superfluous information.
3. **Method synthesis:** Build a model that represents the structure of methods from syntactic information and classification results. Then, map the actions of the specifying part to API calls and inject control structures to form the body; synthesize the method signature.
The first two stages are classification problems. Thus, we apply various machine learning techniques. The first stage is a sequence-to-single-label task, while the second is a typical sequence-to-sequence task. For the first we compare classical machine learning techniques, such as logistic regression and support vector machines, with neural network approaches including the pre-trained language model BERT (Devlin et al., 2019). For the second task we narrow down to neural networks and BERT. A more detailed description of the first two stages may be found in (Weigelt et al., 2020). The implementation of the third stage is a combination of syntactic analysis, knowledge-based techniques and information retrieval. We use semantic role labeling, coreference analysis, and a context model (Weigelt et al., 2017) to infer the semantic model. Afterwards, we synthesize method signatures heuristically and map instructions from the body to API calls using ontology search methods and datatype analysis. Additionally, we inject control structures, which we infer from keywords and syntactic structures. To cope with spontaneous (spoken) language, our approach relies on shallow NLP techniques only.
We carried out a study to examine how laypersons teach new functionality to intelligent systems. The study consists of four scenarios in which a humanoid robot should be taught a new skill: greeting someone, preparing coffee, serving drinks, and setting a table for two. All scenarios take place in a kitchen setting but involve different objects and actions. Subjects were supposed to teach the robot using nothing but natural language descriptions. We told the subjects that a description ideally comprises a declaration of intent to teach a new skill, a name for the skill, and an explanation of intermediate steps. However, we do not force the subjects into predefined wording or sentence structure. Instead, we encouraged them to vary the wording and to 'speak' freely. We also instructed them to imagine that they were standing next to the robot. After the short introduction, we successively presented the scenarios to the subjects. Finally, we requested some personal information in a short questionnaire.
We used the online micro-tasking platform Prolific<sup>1,2</sup>. In less than three days, 870 participants
desc.	w. (total)	w. (unique)
sc. 1 (greet)	795	18,205	566
sc. 2 (coffee)	794	26,005	625
sc. 3 (drinks)	794	33,001	693
sc. 4 (table)	785	31,797	685
total	3,168	109,008	1,469
Table 1: The number of descriptions, words, and unique words per scenario and in the entire dataset.
completed the study. The share of male and female participants is almost equal (50.5% vs. 49.5%); more than  $60\%$  are native English speakers. Most of them  $(70\%)$  had no programming experience at all. An analysis of the dataset revealed that there is barely any difference in the language used by subjects, who are inexperienced in programming, compared to more experienced subjects (except for a few subjects that used a rather technical language). The age of the participants ranges from 18 to 76 with more than half being 30 or younger.
The collected data comprises 3,168 descriptions with more than 109,000 words altogether (1,469 unique words); the dataset statistics are depicted in Table 1. We provide a set of six descriptions from the dataset in Table 13 (Appendix A). A thorough analysis of the dataset revealed that a notable share  $(37\%)$  lacks an explicitly stated intent to teach a skill, albeit we even consider phrases such as "to prepare lunch" as teaching intent. Regarding the semantic structure, we observed that the distinct parts can be clearly separated in almost all cases. However, the respective parts occurred in varying order and are frequently non-continuous.
The data was jointly labeled by two of the authors. We first attached the binary labels teaching and non-teaching. These labels correspond to the classification task from the first stage. Then we add ternary labels (declaration, specification, and miscellaneous) to all words in descriptions that were classified as teaching effort in the first step. This label set is used for the second stage. The distribution of the labels is depicted in Table 2.
Both label sets are unequally distributed, which may cause the machine learning models to overfit in favor of the dominating label. This mainly affects the ternary classification task; the
speech recordings would be more natural. However, from previous studies we learned that subjects more willingly write texts than speak. Besides, the audio quality of recordings is often poor, when subjects use ordinary microphones.
Prolific: https://www.prolific.co/
2We decided to gather textual responses, even though
teaching	binary	ternary
non-teaching	total	declaration	specification	miscellaneous	total
1,998 (.63)	1,170 (.37)	3,168	15,559 (.21)	57,156 (.76)	2,219 (.03)	74,934
Table 2: The distribution of binary and ternary labels in the dataset. The resp. share is given in parenthesis.
random	scenario
Decision Tree	(.893) .903	(.861) .719
Random Forest	(.917) .909	(.893) .374
SVM	(.848) .861	(.870) .426
Naïve Bayes	(.771) .801	(.765) .300
Logistic Regression	(.927) .947	(.891) .719
baseline (ZeroR)	.573	.547
Table 3: Classification accuracy achieved by classical machine learning techniques on validation (in paren.) and test set. The best results are printed in bold type.
label specification distinctly dominates  $(76\%)$  the others. The entire dataset is publicly accessible (open access), including raw data, labeled data, meta-data, and scenario descriptions: http://dx.doi.org/10.21227/zecn-6c61.
3.2 First Stage: Teaching Intentions
The first step of  $f u_{\mathrm{SE}}$  is discovering teaching intents in utterances. An utterance can either be an effort to teach new functionality or merely a description of a sequence of actions. This problem is a typical sequence-to-single-label task, where the words of the utterance are the sequential input and the output is either teaching or non-teaching.
To train, validate, and test classifiers we split up the dataset in two ways. The first is the common approach to randomly split the set in an 80-to-20 ratio, where  $80\%$  of the data is used for training and  $20\%$  for testing. As usual, we again split the training set in 80 parts for training and 20 for validation. However, we felt that this approach does not reflect realistic set-ups, where a model is learned from historical data and then applied to new unseen data, that is semantically related but (potentially) different. Therefore, we introduced an additional so-called scenario-based split in which we separate the data according to the scenarios. We use three of the four scenarios for training and the remaining for testing. Note that we again use an 80-20 split to divide training and validation sets.
We applied classical machine learning and neural network approaches to the task. The classical techniques are: decision trees, random forests,
support vector machines, logistic regression, and Naïve Bayes. As baseline for the classification accuracy we use the so-called Zero-Rule classifier (ZeroR); it always predicts the majority class of the training set, i.e. teaching in this case.
We transform the words to bag-of-words vectors and use tri- and quadrigrams as additional features. The measured accuracy of each classifier on the random and scenario-based data is depicted in Table 3; the validation set accuracy is given in parenthesis and the test set accuracy without.
On the random set all classifiers exceed the baseline. Thus, the (slightly) imbalanced dataset does not seem to affect the classifiers much. Logistic regression performs surprisingly well. However, on the scenario-based split the accuracy of all classifiers decreases drastically. While the accuracies on the validation set remain stable, these classifier techniques are unable to generalize to unseen input. The logistic regression remains the best classifier. However, its accuracy decreases to  $71.9\%$ .
These results reinforced our intuition that deep learning is more appropriate for this task. We implemented a broad range of neural network architectures: artificial neural networks, convolutional networks, and recurrent networks, including LSTMs and GRUs and their bidirectional variants. We experimented with additional layers, which we systematically added to the networks, such as dropout (DO), dense (D), or global max pooling (GMax). We altered all hyper-parameters in reasonable ranges of values<sup>3</sup>. We present only the best performing configurations, i.e. architecture and hyperparameter combinations, in Table 4. Detailed information on the tested hyper-parameter values and further results may be found in Appendices B and C. The words from the input are represented as fastText word embeddings (Bojanowski et al., 2017; Joulin et al., 2017); we use the 300-dimensional embeddings that were trained on the Common Crawl dataset<sup>4</sup> by Facebook Research (Mikolov et al.,
3Note that we do not discuss the influence of varying epoch numbers, since we used early stopping, i.e. the training stops when the validation loss stops decreasing.
4Common Crawl: https://commoncrawl.org/
network architecture	random	scenario
C(128,3), Max(2), C(64,3), GMax, D(10)	(.952).971	(.962).874
C(128,5), Max(2), C(128,5), GMax, D(10)	(.954).966	(.977).862
BiGRU(32), DO(.2), D(64), DO(.2)	(.952).959	(.958).932
BiLSTM(128), D(64)	(.956).959	(.962).919
BERT, 5 epochs	(.973).981	(.991).969
BERT, 10 epochs	(.976).982	(.992).973
BERT, 300 epochs	(.962).982	(.992).977
baseline (Log. Reg.)	(.927).947	(.891).719
2018). Moreover, we use Google's pre-trained language model BERT (base-uncased), which we equipped with a flat binary output layer.
The results attest that deep learning approaches clearly outperform the best classical technique (logistic regression). In particular, the accuracies show smaller differences between random and scenario-based split. This suggests that the classification is more robust. The best accuracy on the scenario test set is achieved by a bidirectional GRU:  $93.2\%$ . Using BERT, the accuracy increases by more than  $4\%$  with a peak at  $97.7\%$  using 300 training epochs. However, the ten-epochs version is a feasible choice, since the accuracy loss is negligible and the training savings are immense.
3.3 Second Stage: Semantic Structures
The second stage, detecting the semantic parts in teaching efforts, is a typical sequence-to-sequence-labeling task with the labels declaration, specification, and miscellaneous. Even though these semantic structures correspond to phrases from a grammatical point of view, we decided to use per-word labels. For this task we only use neural network approaches and BERT. The remaining set-up is similar to the first stage. We again use fastText embeddings and vary the network architectures and hyper-parameters. Except for a ternary output layer, we use the same configuration for BERT as in the first stage.
The results for both, the random and scenario-based split, are reported in Table 5. The bidirectional architectures - be it GRU or LSTM - are
Table 4: Classification accuracy for neural networks on validation (in parenthesis) and test set (best in bold).
network architecture	random	scenario
BiLSTM(128)	(.987) .985	(.981) .976
BiGRU(128)	(.985) .985	(.982) .968
BiLSTM(128), DO(.2)	(.988) .988	(.981) .975
BiLSTM(256), DO(.2)	(.987) .985	(.982) .975
BERT, 5 epochs	(.979) .982	(.979) .965
BERT, 10 epochs	(.983) .985	(.983) .972
BERT, 300 epochs	(.981) .983	(.985) .973
baseline (ZeroR)	.759	.757
Table 5: Classification accuracy achieved by neural networks on validation (in parenthesis) and test set for the second stage. The best results are printed in bold type.
the clear choice for this task; accuracy values are consistently high. Most encouragingly, the decline on the scenario data is negligible (less than  $1\%$ ). Apparently, the models generalize well and are thus resilient to a change in vocabulary. For the second stage the use of BERT is of no advantage; the results even fall behind the best RNN configurations.
3.4 Third Stage: Method Synthesis
During stage three we first transfer the natural language utterances into a model that represents both method definitions and scripts. Afterwards, we synthesize methods (or scripts) from this model. We create a method signature and map instructions in the body to API calls; to synthesize scripts we only map the instructions and inject control structures.
Before we can transfer natural language utterances to the semantic model we must perform a few NLP pre-processing steps that enrich the input with syntactic and semantic information. To obtain parts of speech (PoS), we apply a joint tagging approach; we consolidate the PoS tags produced by the Stanford Log-linear Part-Of-Speech Tagger (Toutanova et al., 2003) and SENNA (Collobert et al., 2011). The Stanford Tagger also provides us with word lemmas. Then we detect individual events in terms of clauses. Since our approach is supposed to cope with spoken language, we are unable to make use of punctuation. Instead, we split the input in a continuous sequence of instructions based on heuristics that make use of PoS tags and keywords. However, the instructions do not necessarily span complete clauses. Thus, we can not apply common parsers. Instead, we use the shallow parser  $BIOS^6$  that provides us with chunks. To obtain semantic roles for each instruction, we again
<sup>5</sup>Again, we only present the best configurations here. For more configurations, refer to Table 16 in Appendix C.
$^{6}$ http://www.surdeanu.info/mihai/bios/
class	description
Thing	Top concept of the ontology
\(\sqsubseteq\) System	(Sub-)Systems (API classes)
\(\sqsubseteq\)Method	System functions (API methods)
\(\sqsubseteq\)Parameter	Parameter names
\(\sqsubseteq\)DataType	Data types used by the system, e.g., int or Graspable
\(\sqsubseteq\) Object	External objects [empty here]
\(\sqsubseteq\)State	Object states [empty here]
Table 6: Domain ontology structure for systems.
class	description
Thing	Top concept of the ontology
\(\sqsubseteq\) Object	Objects in environment
\(\sqsubseteq\) Graspable	Graspable objects, e.g., cup
\(\sqsubseteq\) Openable	Openable objects, e.g., bottle
...
\(\sqsubseteq\) State	Object states, e.g., opened
Table 7: Domain ontology structure for environments.
employ SENNA<sup>7</sup>. Word senses are disambiguated using the tool Babelfy (Moro et al., 2014). Since Babelfy is linked to WordNet (Fellbaum, 1998), we can also make use of synonyms.
We use ontologies to model the target systems, i.e. APIs. An ontology represents the classes, methods, parameters, data types, and values (resp. value ranges), of an API (similar to the ontologies used by Landhauer et al. (2017) and Atzeni and Atzori (2018a,b)). The basic ontology structure is depicted in Table 6. If the system is supposed to interact with an environment, we employ additional ontologies that model the environment including objects and their states (see Table 7). Environment ontologies are merged into system ontologies by copying concepts to the respective placeholders.
To bridge the semantic gap between natural and programming language we introduce a semantic model, as depicted in Figure 2. The model resembles the basic structure of method definitions. However, the leaves are composed of natural language phrases. To determine the phrases that will make up the model elements, we first smooth the classification results provided by the second stage.  $f_{\text{use}}$  maps all phrases of an instruction to the same second-level model element, i.e. either method signature or an instruction of the body. Therefore, we
Figure 2: Exemplary semantic model for an utterance.
unify the second stage classification labels for each instruction using majority decision. Afterwards, we map phrases to leaf elements. Roughly speaking, we use the roles provided by semantic role labeling (SRL) and map predicates to names and arguments to parameters. If we detect a coreference, we substitute the referring expression with the referent, e.g. it with the cup. We also add a lemmatized variant of the phrase and all synonyms. Note that the parameters are a list of phrases.