SlowGuess/ACL_2020_OCR huggingface dataset:
The first step to create method definitions is signature synthesis. To construct a meaningful name, we heuristically clean up the phrase, e.g. remove auxiliary verbs and stop words, and concatenate the remaining words. The parameters are either mapped to data types to infer formal parameters or - if no mapping is to be found - they are attached to the name. For instance, assuming that the declarative instruction is serving wine means,  $fuse$  extracts serve as the first part of the name. Then it tries to map wine to an ontology individual (as discussed later). Assuming it finds the individual RedWineBottle and it is an instance of the concept Graspable in the environment ontology. If the system ontology supports the data type Graspable,  $fuse$  synthesizes the signature serve(serve WHAT: Graspable). Otherwise, the method signature serveWine() is created.
The instructions in the method body are mapped to API calls. Therefore, we first query the ontologies for each leaf element individually. For the queries we use three sets of words we create from the original phrase, the lemmatized version, and the synonyms. We then build the power sets and all permutations of each set, before we concatenate the words to construct a query set. For instance, for the phrase is closed, we produce the query strings: isclosed, closedis, beclose, closebe, closed, is,... The ontology search returns all individuals with a Jaro-Winkler score (Winkler, 1990) above .4 or
<sup>7</sup>SENNA uses the semantic role label set defined in the CoNLL-2004 resp. CoNLL-2005 shared tasks (Carreras and Marquez, 2004, 2005).
individuals	api calls
pre.	recall	F1	avg. rank	pre.	recall	F1	avg. rank
sc. 1	.763	.584 (.776)	.662 (.769)	1.31	.583	.461 (.614)	.515 (.598)	1.47
sc. 2	.783	.742 (.857)	.762 (.818)	1.16	.674	.620 (.713)	.646 (.693)	1.19
sc. 3	.847	.813 (.893)	.830 (.870)	1.16	.672	.645 (.708)	.658 (.690)	1.20
total	.807	.731 (.854)	.767 (.830)	1.20	.653	.590 (.689)	.620 (.670)	1.22
Table 8: The results of the evaluation of the API call mapping for individual elements, i.e. names and parameters, and entire calls. The values in parenthesis denote the results obtained excluding SRL errors.
total	teach	non-teach	API calls
sc. 1	25	18	7	77
sc. 2	25	19	6	97
sc. 3	25	15	10	123
total	75	52	23	297
Table 9: The dataset used to evaluate the third stage.
a fuzzy score $^{8}$  above .15. We decided for these comparatively low thresholds, since we see them as lightweight filters that let pass numerous generally valid candidates. Since an individual may be returned more than once with different scores, we set the score of the individual to the maximum of each of its scores. Afterwards, we construct API calls from the model structure and rate each candidate. We start with the method name candidates. For each candidate we query the ontology for formal parameters. Then, we try to satisfy the parameters with the candidates returned by the individual ontology search. Note that we perform type checking for the parameters (including inheritance if applicable). For instance, for the instruction take the cup we may have found the individual grasp as candidate for a method name and the parameter candidates Mug (type Graspable) and Cupboard (type Location). The ontology indicates that the method grasp has one parameter of type Graspable. Then, the type check ensures that fuse creates the call candidate grasp (Mug) but not grasp (Cupboard). The score is composed of the individual scores of the method names and parameters, the share of mapped words of query string to all words in the query, the ratio of mapped parameters to (expected) formal parameters, and the number of additional (superfluous) parameters. In Appendix D we give a more formal introduction to our scoring approach.
The result of the scoring process is a ranked list of candidates for each instruction. For the time being, we simply use the top-ranked candidates to synthesize the method body. However, re-ranking the candidates based on other semantic resources is promising future work. In a last step, we inject control structures, i.e. conditional branching, various types of loops, and concurrency (Weigelt et al., 2018b,c). The approach is rule-based. We use key phrases, such as in case, until, and at the same time. Proceeding from these anchor points we look for structures that fit into the respective control structure. Here, we apply heuristics on the syntax (based on PoS tags and chunks) and coreference. Utterances that were labeled as non-teaching in the first stage also run through the third stage, except for signature synthesis. Thus, we only construct scripts for this type of utterances.
We determine the quality of the approach for the third stage based on utterances from scenarios one, two, and three, since we used scenario four during development. The assessment is partly manual. Hence, we randomly drew 25 utterances from each scenario to reduce the effort. For each description we used the manual labels of first-stage and second-stage classifications and prepared a gold standard for API calls in the method body. Table 9 depicts the dataset. We did not prepare solutions for the signatures, since plenty of valid solutions are imaginable. Thus, we decided to review the signatures manually afterwards. Of the 52 synthesized method names we assessed eight inappropriate. A name is inappropriate if either the name is off-topic or it contains unrelated terms, e.g. askSpeaker or prepareCoffeeFriend for the scenario How to prepare coffee. Moreover,  $fuse$  correctly mapped 23 parameters without any false positive.
The API ontology used in our setting (household robot) comprises 92 methods, 59 parameters, and 20 data types. To represent the environment (a kitchen) of the robot, we used another ontology
8https://commons.apache.org/proper/ commons-text/apidocs/org/apache/commons/ text/similarity/FuzzyScore.html
4286
individuals	api calls
pre.	recall	F1	avg. rank	pre.	recall	F1	avg. rank
sc. 5	.823	.854 (.944)	.839 (.879)	1.04	.589	.649 (.722)	.617 (.649)	1.04
sc. 6	.920	.876 (.929)	.898 (.925)	1.06	.711	.679 (.721)	.695 (.716)	1.11
total	.886	.869 (.934)	.877 (.909)	1.05	.668	.670 (.721)	.669 (.694)	1.10
Table 10: The results of the end-to-end evaluation, divided in individual elements, i.e. names and parameters, and entire calls. The values in parenthesis denote the results obtained excluding SRL errors.
total	teach	non-teach	API calls
sc. 5	50	44	6	158
sc. 6	50	34	16	315
total	100	78	22	473
Table 11: The end-to-end evaluation dataset.
with 70 objects of six types, and six states. Table 8 details the results for the method body synthesis. Besides precision, recall, and  $\mathrm{F_1}$ , it shows the average rank at which the correct element is to be found. Since the semantic role labeling introduces a vast amount of errors on spoken utterances and our approach heavily depends on it, we also determine recall and  $\mathrm{F_1}$  excluding SRL errors. The results are encouraging. We achieve an  $\mathrm{F_1}$  value of  $76.7\%$  for the individuals and  $62.0\%$  for entire calls; in both cases the precision is slightly ahead of the recall. If we excluded SRL errors, the overall performance increases (about  $7\%$  for individuals and  $5\%$  for calls). Besides the SRL, missing and inappropriate synonyms are a major source of errors. If WordNet lacks a synonym for an important word in the utterance,  $fuse$ 's API mapping may be unable to determine the correct ontology individual. Contrary, if WordNet provides an inappropriate synonym,  $fuse$  may produce an incorrect (superfluous) mapping. In other cases, our language model is unable to capture the semantics of the utterance properly. For example,  $fuse$  creates two method calls for the phrase "make sure you close it": close (...) and make (...) . It may also produce superfluous mappings for explanatory phrases, such as "the machine fills cups", if the second stage did not classify it as miscellaneous. Regarding the composition of API calls (methods plus arguments), the majority of errors is introduced by the arguments. In addition to the afore-mentioned error sources, arguments are often ambiguous. For instance, the phrase "open the door" leaves it up to interpretation, which door was intended to be opened. Even
though  $f_{\mathrm{SE}}$  makes use of an elaborated context model, some ambiguities are impossible to resolve (see section 5). A related issue is the incorrect resolution of coreferences; each mistake leads to a misplaced argument. Most of these error sources can be eliminated, if the pre-processing improves. However, many difficulties simply arise from erroneous or ambiguous descriptions. Still,  $f_{\mathrm{SE}}$  interprets most of them correctly. Most encouragingly, the average rank of the correct element is near 1. Thus, our scoring mechanism succeeds in placing the right elements on top of the list.
To measure the performance of  $f u_{\mathrm{SE}}$  on unseen data, we set up an end-to-end evaluation. We created two new scenarios. They take place in the kitchen setting again, but involve different actions and objects. In the first, subjects are supposed to teach the robot, how to start the dishwasher and in the second, how to prepare cereals. Once more we used Prolific to collect the data and set the number of participants to 110. However, we accepted only 101 submissions, i.e. 202 descriptions. We randomly drew 50 descriptions each. Since the evaluation of the overall approach entails the same output as the third stage, we prepared the gold standard like in subsection 3.4 and used the same ontologies. Table 11 details the dataset used in the end-to-end evaluation. Additionally, we provide five exemplary descriptions from the dataset in Table 14 (Appendix A).
In the end-to-end evaluation our approach synthesized 73 method signatures; five were missed due to an incorrect first-stage classification. Out of 73 synthesized methods we assessed seven to be inappropriate. Additionally, 36 parameters were mapped correctly and no false positives were created. Except for the missing method signatures the results are in line with the third-stage evaluation.
The results for the method body synthesis, as depicted in Table 10, even exceed the previous evaluation. The value of the  $\mathrm{F_1}$  -score is  $87.7\%$  for
4287
pre.	rec.	F1
methods	.924	.884	.904
parameters	.828	.951	.885
API calls	.735	.859	.792
Table 12: Evaluation results for the speech corpus.
individuals and  $66.9\%$  for entire API calls. Again, recall and  $\mathrm{F_1}$  increase, if we exclude SRL errors. However, the effect is smaller here. Moreover, the average rank is also closer to the optimum (1.0) in both cases. Since the first two stages of  $f u_{\mathrm{SE}}$  are based on neural networks, it is difficult to say why the results in the end-to-end evaluation improve. However, we believe the main cause is the introduction of a new test dataset, which has two consequences. First, the models used in the first two stages are learned on all four scenarios instead of three, i.e. the models are trained on a larger dataset, which (presumably) makes them more robust. Second, the new task may be simpler to describe. Consequently, the descriptions comprise simpler wordings and become easier to handle. In summary, the results show that  $f u_{\mathrm{SE}}$  generalizes to different settings – at least in the same domain – and is marginally degraded by error propagation.
To assess how well  $f u_{\mathrm{SE}}$  generalizes to truly spoken utterances we evaluated on another dataset. It is a collection of recordings from multiple recent projects. The setting (instructing a humanoid robot in a kitchen setting) is the same. However, none of the scenarios involved teaching new functionality. Thus, we can only measure  $f u_{\mathrm{SE}}$ 's ability to construct scripts. The descriptions in this dataset comprise control structures to a much larger extent. Altogether the dataset comprises 234 recordings and manual transcriptions. The 108 subjects were mostly under-graduate and graduate students.
On the transcripts we assess the mapping of methods and parameters individually. The results for both and entire calls are depicted in Table 12. Even though the spoken samples comprise a vast number of disfluencies and grammatical flaws,  $f u_{\mathrm{SE}}$  maps more calls correctly. This counter-intuitive effect may be explained by the lower complexity and briefness of the spoken descriptions. Regarding the control structures,  $27.4\%$  were injected correctly. Note that correctly means an appropriate condition plus a block with correct extent. If we lower the standards for condition correctness, the share of correct structures is  $71.23\%$ .
We have presented  $fuse$ , a system for programming in natural language. More precisely, we aim to enable laypersons to teach an intelligent system new functionality with nothing but spoken instructions. Our approach is three-tiered. First, we classify whether a natural language description entails an explicitly stated intent to teach new functionality. If an intent is spotted, we use a second classifier to separate the input into semantically disjoint parts; we identify declarative and specifying parts and filter out superfluous information. Finally, we synthesize method signatures from the declarative and method bodies from the specifying parts. Method bodies contain instructions and control structures. Instructions are mapped to API calls. We implemented the first two steps using classical machine learning and neural networks. Teaching intents are identified with an accuracy of  $97.7\%$  (using BERT). The classification of the semantics is correct in  $97.6\%$  of the cases (using a BiLSTM).
We evaluated  $f u_{\mathrm{SE}}$  on 100 descriptions obtained from a user study. The results are promising;  $f u_{\mathrm{SE}}$  correctly synthesized  $84.6\%$  of the method signatures. The mapping of instructions in the body to API calls achieved an  $F_1$ -score of  $66.9\%$ . In a second evaluation on a speech corpus the  $F_1$ -score for API calls is  $79.2\%$ .
We plan to evaluate  $fuse$  in other domains. It will be interesting to see, if we can reuse (or transfer) the machine learning models as well as the rest of the approach. Future adoptions to  $fuse$  will include the integration of a dialog component. We may query the user in case of ambiguous statements or missing parameters. We have implemented an extensible dialog module and shown that it can be used to resolve ambiguous references, word recognition errors, and missing conditions (Weigelt et al., 2018a). However, we still have to figure out, how to query users properly if an API mapping is ambiguous or parameters are missing. Another improvement concerns the analysis of verb references. Humans often refer to previous actions, which may cause superfluous instructions. We will also implement a sanity check that considers feasibility and meaningfulness of the sequence of actions in the method body. The latter may involve a feedback mechanism via the dialog component. Giving feedback to newly learned method definitions that may be lengthy and therefore unhandy to repeat as a whole is an interesting challenge.
Mattia Atzeni and Maurizio Atzori. 2018a. Towards Semantic Approaches for General-Purpose End-User Development. In 2018 Second IEEE International Conference on Robotic Computing (IRC), pages 369-376.
Mattia Atzeni and Maurizio Atzori. 2018b. Translating Natural Language to Code: An Unsupervised Ontology-Based Approach. In 2018 IEEE First International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pages 1-8.
Amos Azaria, Jayant Krishnamurthy, and Tom M. Mitchell. 2016. Instructable intelligent personal agent. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 2681-2689, Phoenix, Arizona. AAAI Press.
Bruce W. Ballard and Alan W. Biermann. 1979. Programming in Natural Language: "NLC" As a Prototype. In Proceedings of the 1979 Annual Conference (ACM), ACM '79, pages 228-237, New York, NY, USA. ACM.
Andrew Begel. 2004. Spoken Language Support for Software Development. In Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing, VLHCC '04, pages 271-272, USA. IEEE Computer Society.
Andrew Begel and Susan L. Graham. 2005. Spoken Programs. In Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing, VLHCC '05, pages 99-106, USA. IEEE Computer Society.
Alan W. Biermann and Bruce W. Ballard. 1980. Toward Natural Language Computation. Comput. Linguist., 6(2):71-86.
Alan W. Biermann, Bruce W. Ballard, and Anne H. Sigmon. 1983. An Experimental Study of Natural Language Programming. International Journal of Man-Machine Studies, 18(1):71-87.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam. 2017. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 341-350, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. 2019. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, pages 394-410, Phoenix, AZ, USA. Association for Computing Machinery.
Xavier Carreras and Lluis Marquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89-97, Boston, Massachusetts, USA. Association for Computational Linguistics.
Xavier Carreras and Lluis Márquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152-164, Ann Arbor, Michigan. Association for Computational Linguistics.
Bo Chen, Le Sun, and Xianpei Han. 2018. Sequenceto-Action: End-to-End Semantic Graph Generation for Semantic Parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 766-777, Melbourne, Australia. Association for Computational Linguistics.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. J. Mach. Learn. Res., 12:2493-2537.
Alain Désilets, David C. Fox, and Stuart Norton. 2006. VoiceCode: An Innovative Speech Interface for Programming-by-voice. In CHI '06 Extended Abstracts on Human Factors in Computing Systems, CHI EA '06, pages 239-242, New York, NY, USA. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742, Melbourne, Australia. Association for Computational Linguistics.
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press.
Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Mathias Landhauer, Sebastian Weigelt, and Walter F. Tichy. 2017. NLCI: A Natural Language Command Interpreter. Automated Software Engineering, 24(4):839-861.
Lauria Lauria, Guido Bugmann, Theocharis Kyriacou, Johan Bos, and Ewan Klein. 2001. Training personal robots using natural language instruction. IEEE Intelligent Systems, 16(5):38-45.
Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, and Ewan Klein. 2002. Mobile robot programming using natural language. Robotics and Autonomous Systems, 38(3):171-181.
Vu Le, Sumit Gulwani, and Zhendong Su. 2013. SmartSynth: Synthesizing smartphone automation scripts from natural language. In Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '13, pages 193-206, Taipei, Taiwan. Association for Computing Machinery.
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers. 2018. APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 105-114.
Nicholas K. Lincoln and Sandor M. Veres. 2012. Natural Language Programming of Complex Robotic BDI Agents. Journal of Intelligent & Robotic Systems, 71(2):211-230.
Hugo Liu and Henry Lieberman. 2005. Metafor: Visualizing Stories as Code. In IUI '05: Proceedings of the 10th International Conference on Intelligent User Interfaces, pages 305-307. ACM.
Mehdi Manshadi, Daniel Gildea, and James Allen. 2013. Integrating programming by example and natural language programming. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, AAAI'13, pages 661-667, Bellevue, Washington. AAAI Press.