SlowGuess/NAACL_2022_OCR huggingface dataset:
We clipped the negative attribution scores to zero to draw attention to positive attributions that corresponds to alignments. Some resulting matrices from the test set are plotted as heatmaps and shown in Figure 3. The matrix on the left shows the alignment matrix for the input invocation 'Sources script incl.sh in folder where current script is located'. The corresponding command is source  $(\text{dirname $0}) / \text{incl.sh}$ . This is a Bash command substitution pattern. The inner command first finds the directory name of the directory containing the currently running script with the dirname utility. It then executes the incl.sh file in that directory with source utility. One can observe that the bash command source aligns with the invocation segment sources script, and the token  $(\text{dirname aligns with segments in folder and where from the invocation.}$
Similarly, Figure 3 (matrix on the right) shows the alignment matrix for input invocation 'Recursively change owner and group to "$JBOSS_AS_USER" of "$JBOSS_AS_DIR", with the corresponding command chown -R $JBOSS_AS_USER: $JBOSS_AS_USER $JBOSS_AS_DIR. The alignment matrix depicts the correspondence between the invocation segment recursively and the command flag -R. In this instance, we also see that some command components are erroneously attributed. For instance
<sup>4</sup>We use the Integrated Gradient implementation provided by the Captum library - https://www.github.com/pytorch/captum
Command Component	Total Count	Attributed Correctly
Utility	129	94 (72.87%)
Flag	160	110 (68.75%)
Argument	244	84 (34.43%)
Table 4: Results from the evaluation of attributions produced by Integrated Gradients for 100 random examples from the test set.
$-\mathbb{R}$  is also attributed to REGEX of REGEX.
We sample 100 examples from the test set and manually evaluate the attribution matrices produced by the IG method. Attribution for a command component is labelled as correct if one can look at only the positively attributed invocation segments to determine the presence of the command component in the output command. For instance, consider the attribution matrix on the right in Figure 3. Here, the counts of correctly attributed utilities, flags and arguments will be 0,1 and 0 respectively. Only the flag  $-\mathbb{R}$  can be figured out from the positively attributed input segment recursively. There is a positive attribution on the invocation segment owner and group. However, in the absence of a positive attribution on change, one can not conclude that the utility chown would be used.
The results of the attribution analysis are shown in Table 4. We observe that utilities and flags have higher attribution accuracy than the arguments. This is due to the preprocessing which normalizes all file paths and regex expressions. It is also observed that sometimes multiple utilities are needed to perform the task in the invocation. There is little alignment between the utilities and invocation segments in such cases. For instance, when utilities like sed and awk are connected using the pipe operator (|) to modify the output of other utilities. Such implicit need for some utilities results in incorrect attribution by SIT. Similarly, it is observed that flags like -and and -or cannot be explicitly aligned to the invocation segments.
From the attribution analysis, we find that the proposed architecture is indeed able to capture the synchronous structure between natural language segments and Bash command components.
Early works on semantic parsing explored meaning representations like first-order logic, lambda cal
<sup>5</sup>Attribution matrices for the sampled instances are available at https://github.com/Shikhar-S/Segmented-Invocation-Transformer/blob/main/jup Notebook/attribute_viz.ipynb
culus enhanced first-order logic (Carpenter, 1997), database query languages and operated on handcrafted rules (Johnson, 1984). These were followed by statistical models that were able to learn rules from input-output parallel data (Thompson, 2003, Zettlemoyer and Collins, 2007, Kwiatkowski et al., 2010).
Recently, there have been many advancements in generating high-level programming languages. Dong and Lapata (2016) and Ling et al. (2016) propose general attention based encoder-decoder style methods for semantic parsing. Rabinovich et al. (2017) propose Abstract Syntax Networks with a dynamically determined modular decoder structure that parallels the structure of the output tree. Yin and Neubig (2017) propose an architecture enhanced by a grammar model that explicitly captures the target language syntax as prior knowledge. Most of the innovations in this area utilize recurrent neural networks (RNN) for modelling natural language input. The method proposed in this work enhances the Transformer encoder with constituency parsing information.
For natural language to Bash, in particular, Lin et al. (2018) created a dataset and proposed an encoder-decoder based architecture. Gros (2019) explore several sequence to sequence models, Abstract Syntax Networks and Nearest Neighbor based models for this task on a custom dataset. Agarwal et al. (2020) proposed a command-line AI assistant for this task. Agarwal et al. (2021) organized a contest in NeurIPS 2020 for natural language to Bash translation and provided a report on the state of the art architectures developed in the contest. Bharadwaj and Shevade (2021) explored the use of Linux manual pages and Abstract Syntax Tree for developing an explainable natural language to Bash translation system. In contrast to these methods, our work explores the synchronous structure of this problem and uses the constituency tree to better model natural language input.
Constituency parse tree of natural language has been used in earlier machine translation and semantic parsing literature. Nguyen et al. (2019) note that the Transformer (Vaswani et al., 2017) struggles to encode hierarchical structures and propose a hierarchical accumulation mechanism that utilizes constituency parse tree to capture this structure for neural machine translation. They achieve this by adding additional parameters that capture the constituency structure of sentences. Our method, in
contrast, uses constituency tree information in a parameter efficient manner. Xu et al. (2018) construct a syntactic graph from constituency and dependency parse tree and employ a graph to sequence neural network using an RNN decoder. They report improvement over the sequence to sequence model proposed by Dong and Lapata (2016) showing that additional syntactic information helps in semantic parsing. We use a Transformer and show that segmentation structure aids in natural language to Bash translation.
We propose a method that utilizes information from the constituency tree to better model the structure of natural language to Bash task. Our experiments on the NLC2CMD data show that incorporating the problem structure in the model architecture improves both performance and parameter efficiency. We also run inference time benchmarks and find that the proposed method is faster. Attribution analysis is also conducted to analyze the method.
In this work, we focus on Bash as the meaning representation and identify the structure for natural language to Bash translation. However, we expect a similar structure for other meaning representations like SQL. Applying our method to natural language to SQL task is left for future work. Our method relies on Stanford CoreNLP parser (Manning et al., 2014) for constituency parsing. This is a bottleneck for fully utilizing the efficiency of our approach. It will be interesting to test faster constituency parsers (Zhang et al., 2020a, Zhang et al., 2020b) which can parse around 1K sentences per second.
Mayank Agarwal, Jorge J Barroso, Tathagata Chakraborti, Eli M Dow, Kshitij Fadnis, Borja Godoy, Madhavan Pallan, and Kartik Talamadupula. 2020. Project clai: Instrumenting the command line as a new environment for ai agents. arXiv preprint arXiv:2002.00762.
Mayank Agarwal, Tathagata Chakraborti, Quchen Fu, David Gros, Xi Victoria Lin, Jaron Maene, Kartik Talamadupula, Zhongwei Teng, and Jules White. 2021. Neurips 2020 nlc2cmd competition: Translating natural language to bash commands. In Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 302-324. PMLR.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Shikhar Bharadwaj and Shirish Shevade. 2021. Explainable natural language to bash translation using abstract syntax tree. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 258-267, Online. Association for Computational Linguistics.
Bob Carpenter. 1997. Type-logical semantics. MIT press.
Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on information theory, 2(3):113-124.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33-43, Berlin, Germany. Association for Computational Linguistics.
David Gros. 2019. Ainix: An open platform for natural language interfaces to shell commands. Undergraduate Honors Thesis.
Tim Johnson. 1984. Natural language computing: the commercial applications. The Knowledge Engineering Review, 1(3):11-23.
Aishwarya Kamath and Rajarshi Das. 2019. A survey on semantic parsing. In *Automated Knowledge Base Construction* (AKBC).
Hyeonji Kim, Byeong-Hoon So, Wook-Shin Han, and Hongrae Lee. 2020. Natural language to sql: Where are we today? Proc. VLDB Endow., 13(10):1737-1750.
Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic ccg grammars from logical form with higher-order unification. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 1223-1233.
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočisky, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 599-609, Berlin, Germany. Association for Computational Linguistics.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55-60.
Xuan-Phi Nguyen, Shafiq Joty, Steven Hoi, and Richard Socher. 2019. Tree-structured attention with hierarchical accumulation. In International Conference on Learning Representations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139-1149.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 3319-3328. JMLR.org.
Cynthia Thompson. 2003. Acquiring word-meaning mappings for natural language interfaces. Journal of Artificial Intelligence Research, 18:1-44.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pretrained encoder-decoder models for code understanding and generation. In Proceedings of the 2021
Conference on Empirical Methods in Natural Language Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Kun Xu, Lingfei Wu, Zhiguo Wang, Mo Yu, Liwei Chen, and Vadim Sheinin. 2018. Exploiting rich syntactic information for semantic parsing with graph-to-sequence model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 918-924.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Squizer: Query synthesis from natural language. Proc. ACM Program. Lang., 1(OOPSLA).
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 440-450. Association for Computational Linguistics.
Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed ccg grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 678-687.
Yu Zhang, Zhenghua Li, and Min Zhang. 2020a. Efficient second-order TreeCRF for neural dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3295-3305, Online. Association for Computational Linguistics.
Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020b. Fast and accurate neural CRF constituency parsing. In Proceedings of IJCAI, pages 4046-4053.