SlowGuess/ACL_2020_OCR huggingface dataset:
Irena Markievicz, Minija Tamosiunaite, Daiva Vitkute-Adzgauskiene, Jurgita Kapociute-Dzikiene, Rita Valterye, and Tomas Krilavicius. 2017. Reading Comprehension of Natural Language Instructions by Robots. In Beyond Databases, Architectures and Structures. Towards Efficient Solutions for Data Analysis and Knowledge Representation, pages 288-301. Springer.
Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 2772-2778, Phoenix, Arizona. AAAI Press.
Rada Mihalcea, Hugo Liu, and Henry Lieberman. 2006. NLP (Natural Language Processing) for NLP (Natural Language Programming). In Proceedings of the 7th International Conference on Computational Linguistics and Intelligent Text Processing, CICling'06, pages 319-330, Berlin, Heidelberg. Springer-Verlag.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231-244.
David Price, Ellen Riloff, Joseph Zachary, and Brandon Harvey. 2000. Natural Java: A Natural Language Interface for Programming in Java. In Proceedings of the 5th International Conference on Intelligent User Interfaces, IUI '00, pages 207-211, New Orleans, Louisiana, USA. ACM.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract Syntax Networks for Code Generation and Semantic Parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1139-1149. Association for Computational Linguistics.
Lanbo She, Yu Cheng, Joyce Y. Chai, Yunyi Jia, Shaohua Yang, and Ning Xi. 2014. Teaching Robots New Actions through Natural Language Instructions. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 868-873, Edinburgh, UK. IEEE.
Alane Suhr and Yoav Artzi. 2018. Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2072-2082, Melbourne, Australia. Association for Computational Linguistics.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 252-259.
Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 929-938, Vancouver, Canada. Association for Computational Linguistics.
Sebastian Weigelt, Tobias Hey, and Mathias Landhauer. 2018a. Integrating a Dialog Component into a Framework for Spoken Language Understanding. In Proceedings of the 6th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering, RAISE '18, pages 1-7, New York, NY, USA. ACM.
Sebastian Weigelt, Tobias Hey, and Vanessa Steurer. 2018b. Detection of Conditionals in Spoken Utterances. In 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pages 85-92.
Sebastian Weigelt, Tobias Hey, and Vanessa Steurer. 2018c. Detection of Control Structures in Spoken Utterances. International Journal of Semantic Computing, 12(3):335-360.
Sebastian Weigelt, Tobias Hey, and Walter F. Tichy. 2017. Context Model Acquisition from Spoken Utterances. In Proceedings of The 29th International Conference on Software Engineering & Knowledge Engineering, pages 201-206, Pittsburgh, PA.
Sebastian Weigelt, Vanessa Steurer, Tobias Hey, and Walter F. Tichy. 2020. Roger that! Learning How Laypersons Teach New Functions to Intelligent Systems. In 2020 IEEE 14th International Conference on Semantic Computing (ICSC), pages 93-100.
William E. Winkler. 1990. String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage.
Terry Winograd. 1972. Understanding natural language. Cognitive Psychology, 3(1):1-191.
The dataset includes descriptions of varying quality. Some texts have syntactical flaws such as typos and grammar mistakes. They also vary in terms of descriptiveness and style; the latter ranges from full sentences to notes. Table 13 shows six examples from the preliminary study (scenarios one to four) and Table 14 five examples from the end-to-end evaluation (scenarios five and six). Most of the descriptions contain errors. For instance, description 2180 contains typos, such as "ring some beverage".
B Architectures and Hyper-parameters
We applied a broad range of machine learning approaches to the classification tasks. Table 15 shows the types, architectures and hyper-parameters we tested in the process. We also experimented with self-trained and pre-trained fastText embeddings.
C Configurations and Results
Table 16 shows representative configurations for the first stage of  $f u_{\mathrm{SE}}$  (binary classification); for neural networks we altered the hyper-parameters systematically to give an intuition of the effects. There are general trends. Classifiers perform better on randomly split data, a batch size of 100 is better than 300, and pre-trained embeddings outperform the self-trained in almost all cases. Overall, BERT-based classifiers achieve the best results. However, some neural network configurations come close (e.g.  $\mathrm{RNN}_{6.0}$ ); classical machine learning techniques are inadequate. For the second stage (ternary classification) we show interesting results in Table 17. The trends are as follows. The preferable batch size is 32, pre-trained embeddings again outperform the self-trained, and RNNs are best.
D Call Candidate Scoring
In subsection 3.4 we only discuss the rationale behind our call candidate scoring mechanism. Subsequently, we give a formal introduction. A call candidate is an API method with arguments (extracted from the natural language input). The arguments are of either primitive, composite (strings or enumerations), or previously defined types (e.g. objects from the environment). The arguments adhere to the formal definition of the API method. For each call candidate  $c f u_{\mathrm{SE}}$  calculates the score  $S(c)$  as follows:
$$
S (c) = \phi * P (c) * S _ {M} (c) + (1 - \phi) * W S _ {P} (c) \tag {1}
$$
The score is composed of two components: the method score  $S_{M}(c)$  and the weighted parameter score  $W S_{P}(c)$ . The impact of the latter on the final score can be adjusted with the weight  $\phi$ . Further,  $S_{M}(c)$  is scaled by the perfect match bonus  $P(c)$ :
$$
P (c) = \left\{ \begin{array}{l l} \tau & M (c) > 0. 9 \\ 1 & \text {o t h e r w i s e} \end{array} \right. \tag {2}
$$
The perfect match bonus  $P(c)$  allows us to prefer call candidates with a method name score  $M(c)$  above 0.9. The scaling factor  $\tau$  is configurable  $(\tau \geq 1)$ . The method score  $S_{M}(c)$  is computed as follows:
$$
S _ {M} (c) = M (c) - \frac {\beta}{\left| I _ {A} (c) \right|} * \left(1 - \frac {\left| I _ {F} (c) \right|}{\left| I _ {A} (c) \right|}\right) \tag {3}
$$
The method name score  $M(c)$  is the maximal similarity of the natural language chunk that represents the action (or event) and the (API) method name. We use Jaro-Winkler and fuzzy score as similarity measures. To obtain the method score  $S_{M}(c)$ , the method name score  $M(c)$  is reduced by a subtrahend that indicates how well the method name represents the words in the original natural language chunk. The subtrahend is composed of two factors. The second is one minus the fraction of words in the chunk that can be found in the method name and the total amount of words in the chunk; i.e., this factor is the share of unmapped words. The other factor scales it by a configurable parameter  $\beta$ , which is divided by length of the chunk. The rationale behind this is as follows. In short chunks each word is important. Therefore, unmapped words are strongly penalized. With an increasing number of words in the chunk, it is increasingly unlikely to map all words. However, in longer chunks many words are semantically irrelevant. Therefore, we reduce the subtrahend with the length of the chunk.
The weighted parameter score  $W S_{P}(c)$  in Equation 1 is calculated as follows:
$$
W S _ {P} (c) = S _ {P} (c) - \omega * P e n (c) \tag {4}
$$
The score is composed of the parameter score  $S_P(c)$  and a penalty value  $Pen(c)$ ; the latter is weighted by the configurable factor  $\omega$ . The parameter score  $S_P(c)$  is calculated as follows:
$$
S _ {P} (c) = \sum P _ {i} (c) * \frac {\left| P _ {M} \right|}{\left| P _ {O} (c) \right|} \tag {5}
$$
ID
scen. text
302	1	Look directly at the person. Wave your hand. Say ‘hello’.
1000	2	You have to place the cup under the dispenser and press the red button to make coffee.
1346	2	Making coffee means you have to press the red button, put a cup underneath the hole and then pouring the coffee that comes out into your cup
2180	3	To ring a beverage, open the fridge and select one of te beverages inside, pour it into one of the glasses on the kitchen counter and hand the glass over to the person.
2511	4	collect cutlery from cupboard, bring them to the table and place down neatly
2577	4	To set the table for two, Go to the cupboard and take two of each; plates, glasses, knives, and forks. Take them to the kitchen table and set two individual places.
Table 13: Six exemplary submissions taken from the preliminary study dataset (scenarios one to four).
ID	scen.	text
E_10	5	Hey, Amar. We're going to start the dishwasher so what we have to do is first make sure the dishwasher is closed and then press the blue button twice to start the dishwasher.
E_79	5	Hi Armar. Turning on the Dishwasher means you have to go to the dishwasher. Close the dishwasher, then press the blue button 2 times.
E_81	5	Hi Armar, to use the dishwasher you need to check first if it is closed, if not, close it by pushing the front door. If it is closed, look for the blue button on the dishwasher. Once you find it, press it a first time and then a second time. That's how you start the dishwasher.
E_117	6	hi armar, you get your cereal ready you need to go to the fridge and open the door by pulling it. you will find the milk bottle inside the fridge door. lift it out and carry it to the kitchen table. place it next to your bowl and cereal box. start by filling the bowl with your cereal then pour in some milk.
E_158	6	Hi Armar, you have to go to the fridge, open it, grab the milk, close it and carry the milk to the kitchen table. Then place it next to the bowl and the cereal box. Fill the bowl with the cereals and then pour the mil in the bowl. That is how you prepare some cereals
Table 14: Five exemplary submissions taken from the end-to-end evaluation dataset (scenarios five and six).
$P_{M}$  is the set of all parameters  $p_i$  (extracted from the natural language input) that were mapped to formal method parameters. Each  $p_i$  has a similarity score  $(P_i(c))$ . Thus,  $S_P(c)$  is the sum of all similarity scores of mapped parameters multiplied with the share of mapped  $(P_M)$  and expected formal parameters as defined in the ontology  $(P_O(c))$ . To calculate  $WS_P(c)$  (see Equation 4),  $S_P(c)$  is reduced by the penalty value  $Pen(c)$  that is calculated as follows:
$$
P e n (c) = \frac {\left| P _ {E} \right| - \left| P _ {M} \right|}{\left| P _ {E} \right|} \tag {6}
$$
$P_{E}$  is the set of parameters that were extracted from natural language input (see Figure 2). Thus,  $Pen(c)$  is the number of parameters in the input that were not mapped to a formal method parameter, normalized by the total amount of extracted (natural language) parameters.
For the evaluation of the third stage of  $f u_{\mathrm{SE}}$  and the end-to-end-evaluation we set the method score weight  $\phi$  to 0.6, the perfect match multiplier  $\tau$  to 1.5, the search string coverage weight  $\beta$  to 0.5, and the penalty factor  $\omega$  to 0.3. We determined all values empirically with the help of examples from scenario 4.
types	architect.	additional layers	number of units	epochs	batch sizes	dropout values	learning rates
ANN	Flatten (Flat), GMax, Dense (D), Dropout(DO)	10, 32, 50, 64, 100, 128, 200, 256	binary: 300, 500, 1000 ternary: 50, 100, 300	binary: 100, 300 ternary: 32, 64, 100, 300	0.2, 0.3, 0.4	0.001, 0.0005
CNN	CONV	Max, GMax, Dense (D), Dropout(DO)
RNN	LSTM GRU BiLSTM BiGRU	Dense (D), Dropout (DO)
BERT	Flatten (Flat)	5, 10, 300	32	0.00002
Table 15: Overview of the types, architectures, and hyper-parameters of neural networks used in the two classification tasks (step one and two of  $f u S E$ ).
name	additional layers	batch size	random	scenario
self-trained	fastText	self-trained	fastText
\(ANN_{1.0}\)	Flat, D(10)	100	(.907).911	(.874).887	(.918).759	(.897).722
\(ANN_{2.0}\)	Flat, D(100)	100	(.916).914	(.846).867	(.905).781	(.874).715
\(ANN_{2.1}\)	Flat, D(100)	300	(.921).922	(.844).870	(.922).732	(.863).577
\(ANN_{3.0}\)	GMax, D(10)	100	(.876).887	(.872).902	(.907).766	(.905).542
\(ANN_{3.1}\)	GMax, D(100)	100	(.899).896	(.879).896	(.893).668	(.918).674
\(ANN_{3.2}\)	GMax, D(100)	300	(.888).889	(.877).897	(.895).676	(.908).428
\(CNN_{1.0}\)	Conv(128, 3), GMax, D(10)	100	(.947).966	(.954).963	(.962).765	(.966).854
\(CNN_{1.1}\)	Conv(128, 5), GMax, D(10)	100	(.947).971	(.930).965	(.973).743	(.973).776
\(CNN_{1.2}\)	Conv(128, 7), GMax, D(10)	100	(.952).966	(.943).962	(.973).775	(.970).897
\(CNN_{2.0}\)	Conv(128, 3), Max(2), Conv(64, 3), GMax, D(10)	100	(.952).959	(.952).971	(.968).855	(.962).874
\(CNN_{2.1}\)	Conv(128, 5), Max(2), Conv(64, 5), GMax, D(10)	100	(.949).972	(.952).966	(.969).850	(.975).859
\(CNN_{2.2}\)	Conv(128, 5), Max(2), Conv(128, 5), GMax, D(10)	100	(.952).964	(.954).966	(.973).862	(.977).862
\(CNN_{2.3}\)	Conv(128, 5), Max(2), Conv(128, 5), GMax, D(10)	300	(.952).953	(.947).965	(.973).783	(.971).901
\(CNN_{2.4}\)	Conv(128, 5), Max(5), Conv(128, 5), GMax, D(10)	100	(.956).958	(.952).959	(.962).901	(.973).801
\(RNN_{1.0}\)	GRU(128)	100	(.560).625	(.562).625	(.477).299	(.519).702
\(RNN_{1.1}\)	GRU(128), D(100)	100	(.562).625	(.562).625	(.519).702	(.519).702
\(RNN_{2.0}\)	BiGRU(32), DO(0.2), D(64), DO(0.2)	100	(.947).944	(.952).959	(.954).911	(.958).932
\(RNN_{3.0}\)	LSTM(64)	100	(.566).631	(.568).638	(.519).702	(.519).702
\(RNN_{3.1}\)	LSTM(128)	100	(.570).625	(.654).738	(.519).702	(.519).702
\(RNN_{4.0}\)	LSTM(128), D(100)	100	(.562).625	(.562).625	(.519).702	(.519).702
\(RNN_{4.1}\)	LSTM(128), D(100)	300	(.562).625	(.567).633	(.519).702	(.519).702
\(RNN_{5.0}\)	BiLSTM(64), DO(0.2), D(64), DO(0.2)	100	(.947).955	(.949).955	(.956).896	(.962).916
\(RNN_{5.1}\)	BiLSTM(64), DO(0.2), D(64), DO(0.2)	300	(.929).919	(.954).949	(.945).650	(.966).872
\(RNN_{5.2}\)	BiLSTM(64), DO(0.3), D(200), D(100)	100	(.941).947	(.947).949	(.947).884	(.956).911
\(RNN_{6.0}\)	BiLSTM(128), D(64)	100	(.951).955	(.956).959	(.960).927	(.962).919
\(RNN_{6.1}\)	BiLSTM(128), D(64), D(32)	100	(.945).962	(.947).955	(.950).919	(.966).898
\(RNN_{7.0}\)	BiLSTM(128), D(100), DO(0.3), D(50)	100	(.936).937	(.945).941	(.937).922	(.954).917
\(RNN_{7.1}\)	BiLSTM(128), D(100), DO(0.3), D(50)	300	(.934).934	(.938).947	(.937).704	(.950).907
\(RNN_{8.0}\)	BiLSTM(256), D(128)	100	(.952).944	(.945).952	(.954).843	(.962).912
\(BERT_1\)	5 epochs	32	(.973).981	(.991).969
\(BERT_2\)	10 epochs	32	(.976).982	(.992).973
\(BERT_3\)	300 epochs	32	(.962).982	(.992).977
Decision Tree	(.893).903	(.861).719
Random Forest	(.917).909	(.893).374
Support Vector Machine	(.848).861	(.870).426
Naïve Bayes	(.771).801	(.765).300
Logistic Regression	(.927).947	(.891).719
baseline (ZeroR)	.573	.547
Table 16: Classification accuracy obtained on the validation (in parenthesis) and the test set for the first stage (binary classification). The best results (per classifier category) are printed in bold type. The basic structure of each neural network includes an embedding layer and an output layer (dense layer).
name	additional layers	batch size	random	scenario
self-trained	fastText	self-trained	fastText
\(ANN_{1.0}\)	-	32	(.851).855	(.851).856	(.850).779	(.851).826
\(ANN_{1.1}\)	-	100	(.851).849	(.852).849	(.849).746	(.851).826
\(ANN_{2.0}\)	D(10)	32	(.848).857	(.852).849	(.850).825	(.851).826
\(ANN_{2.1}\)	D(100)	32	(.853).856	(.853).848	(.851).822	(.851).827
\(RNN_{1.0}\)	LSTM(64)	32	(.977).976	(.979).978	(.971).960	(.975).966
\(RNN_{1.1}\)	LSTM(64)	100	(.973).972	(.978).975	(.969).952	(.974).964
\(RNN_{1.2}\)	LSTM(128)	32	(.974).976	(.978).977	(.973).960	(.973).964
\(RNN_{1.3}\)	LSTM(128)	100	(.974).975	(.978).977	(.970).962	(.971).965
\(RNN_{1.4}\)	LSTM(128)	300	(.973).973	(.977).974	(.968).954	(.972).961
\(RNN_{2.0}\)	LSTM(128), DO(0.2)	32	(.976).977	(.977).977	(.970).960	(.973).966
\(RNN_{2.1}\)	LSTM(128), DO(0.4)	32	(.976).977	(.979).979	(.971).959	(.974).967
\(RNN_{3.0}\)	LSTM(128), D(64)	32	(.973).972	(.977).976	(.970).955	(.971).963
\(RNN_{4.0}\)	BiLSTM(64)	32	(.987).984	(.987).985	(.982).949	(.981).972
\(RNN_{4.1}\)	BiLSTM(64)	100	(.981).980	(.986).984	(.979).960	(.982).967
\(RNN_{4.2}\)	BiLSTM(128)	32	(.986).983	(.987).985	(.983).960	(.981).976
\(RNN_{4.3}\)	BiLSTM(128)	64	(.984).983	(.987).984	(.979).952	(.982).973
\(RNN_{4.4}\)	BiLSTM(128)	100	(.985).983	(.986).984	(.983).960	(.981).969
\(RNN_{4.5}\)	BiLSTM(128)	300	(.983).982	(.985).984	(.977).956	(.980).968
\(RNN_{5.0}\)	BiLSTM(128), D(64)	32	(.980).983	(.985).984	(.973).960	(.979).965
\(RNN_{6.0}\)	BiLSTM(128), D(100), DO(0.3), D(50)	32	(.982).982	(.982).985	(.978).955	(.981).968
\(RNN_{7.0}\)	BiLSTM(128), DO(0.2)	32	(.985).984	(.988).988	(.982).958	(.981).975
\(RNN_{7.1}\)	BiLSTM(128), DO(0.4)	32	(.985).986	(.986).986	(.980).961	(.980).973
\(RNN_{7.2}\)	BiLSTM(256), DO(0.2)	32	(.986).984	(.987).985	(.982).964	(.982).975
\(RNN_{8.0}\)	BiGRU(128)	32	(.984).984	(.985).985	(.976).955	(.982).968
\(BERT_1\)	5 epochs	32	(.979).982	(.979).965
\(BERT_2\)	10 epochs	32	(.983).985	(.983).972
\(BERT_3\)	300 epochs	32	(.981).983	(.985).973
baseline (ZeroR)	.759	.757
Table 17: Classification accuracy obtained on the validation (in parenthesis) and the test set for the second stage (ternary classification). The best results are printed in bold type. The basic structure of each model includes an embedding layer and an output layer (dense layer).
4295