SlowGuess/NAACL_2022_OCR huggingface dataset:
Shikhar Bharadwaj and Shirish Shevade
Department of Computer Science and Automation,
Indian Institute of Science, Bangalore, KA 560012, India
{shikharb,shirish}@iisc.ac.in
Bash is a Unix command language used for interacting with the Operating System. Recent works on natural language to Bash translation have made significant advances, but none of the previous methods utilize the problem's inherent structure. We identify this structure and propose a Segmented Invocation Transformer (SIT) that utilizes the information from the constituency parse tree of the natural language text. Our method is motivated by the alignment between segments in the natural language text and Bash command components. Incorporating the structure in the modelling improves the performance of the model. Since such systems must be universally accessible, we benchmark the inference times on a CPU rather than a GPU. We observe a  $1.8\mathrm{x}$  improvement in the inference time and a  $5\mathrm{x}$  reduction in model parameters. Attribution analysis using Integrated Gradients reveals that the proposed method can capture the problem structure.
Semantic parsing is one of the central tasks for natural language understanding (NLU). It is defined as the task of generating meaning representations from natural language utterances (Kamath and Das, 2019). Previous works (Yin and Neubig, 2017, Yaghmazadeh et al., 2017, Kim et al., 2020, Agarwal et al., 2021) have used high level languages such as Python, SQL and Bash as meaning representations. This work focuses on generating Bash commands from natural language descriptions of command-line tasks.
Besides being an essential task for NLU, semantic parsing into a high-level language also has real-world applications such as helping developers write programs and making programming universally accessible. The command-line interface has been regarded as an invaluable tool due to its expressiveness, efficiency and extensibility (Agarwal et al.,
2021). However, it has a learning curve and requires domain knowledge. An interface with the computer using natural language, on the other hand, remedies these issues. One need not remember the syntax of hundreds of Bash utilities, and instead, one can specify the task in natural language. Such an interface that uses natural language such as English for specifying command-line tasks makes computing accessible to people with little domain knowledge. Therefore, developing a system to generate Bash commands from English is worth one's efforts.
Previous works on this semantic parsing task (Lin et al., 2018, Gros, 2019, Agarwal et al., 2020, Agarwal et al., 2021, Bharadwaj and Shevade, 2021) use various encoder-decoder style architectures. These methods consider the natural language component as a sequence of tokens without utilizing the inherent structure for this problem. The method proposed in this work utilizes the information from the constituency parse tree of the natural language to incorporate this problem structure into the modelling process. Our approach is based on the observation that natural language invocations are complex and can be broken down into simpler segments that align with the Bash command components (utilities, flags and arguments). We incorporate this observation in our method to provide an inductive bias to the Transformer model (Vaswani et al., 2017), making the search space of solutions more aligned with the task at hand. The models are evaluated on the NL2Bash dataset (Lin et al., 2018) obtained from the NeurIPS 2020 Natural Language Context to Command (NLC2CMD) contest (Agarwal et al., 2021). The proposed method outperforms the winning solution from the NLC2CMD contest in terms of generation accuracy while also achieving a speedup of  $1.8\mathrm{x}$  and reducing the parameter count by  $5\mathrm{x}$  over it. It also performs better than models like T5 (Raffel et al., 2019) and
Human Language Technologies, pages 3159 - 3168
CodeT5 (Wang et al., 2021) which are trained on a large amount of data and then fine-tuned on the dataset under study. Our code is available at https://github.com/Shikhar-S/Segmented-Invocation-Transformer Our main contributions are the following:
- We identify the structure for natural language to Bash generation task and propose a constituency tree based method for incorporating the structure in the Transformer (Vaswani et al., 2017) framework. The proposed modification improves the performance of the Transformer on this task.
- We benchmark the Transformer against the proposed architecture. Results show a reduction in inference time and the number of parameters.
- We conduct attribution analysis using Integrated Gradients (Sundararajan et al., 2017) to analyze the proposed method's workings.
First, we formally describe the problem statement. Section 2 describes the structure for the problem, and Section 3 describes our approach to model the structure and expected gains in the inference time via a complexity analysis of the decoding phase. Section 4 describes the dataset used and its preprocessing. In Section 5, we describe the experiments conducted for checking the correctness and efficiency of our approach and analyzing the results. Section 6 compares our work with other related works. Finally, in Section 7 we conclude and discuss some directions for future work.
Problem Statement. Let  $I$  be the set of all natural language invocations,  $C$  be the set of all Bash
Figure 1: Segmenting Invocation: First, the raw invocation is normalized to remove patterns and file paths. This normalized invocation is then parsed to obtain a constituency tree. Then, the tree is cut at a threshold height to create subtrees. The leaves of each subtree form tokens in the segments.
commands and  $\mathcal{D} \coloneqq \{(nlc, c)\}$  be a parallel natural language invocation-Bash command dataset, where  $nlc \in I$  and  $c \in C$ .
The task is to design an algorithm that, given an invocation  $nlc \in I$  and dataset  $\mathcal{D}$ , outputs a set of Bash command-confidence pairs  $(\hat{c},\delta)$  such that
-  $\hat{c} \in C$  is the predicted Bash command that performs the task specified in  $nlc$ , and
-  $\delta \in [0,1]$  is the associated confidence score.
For example,  
nlc = Delete files with inode number specified by REGEX under current directory.
$$
c = \text {f i n d} . - \text {i n u m R E G E X} - \text {e x e c r m} - \mathrm {i} \{\} \backslash ;
$$
2 Problem Structure
On conducting a manual analysis of a few examples from the training data, it was observed that natural language invocations are often complex and can be broken down into simpler descriptive segments needed for the task. These segments would often map directly to a Bash command component- a utility, a flag or an argument. For instance, consider
| Delete | files | with | inode number | specified |
| --- | --- | --- |
| by REGEX | under current directory. |
BashCommand
find . -inum REGEX -exec rm -i {} ;
Invocation segment	Bash command component
Delete	rm
inode number	-inum
by REGEX	REGEX
under current directory	.
Table 1: Segments from natural language invocation that align with Bash command components (utilities, flags and arguments).
Table 1 lists the segments of the invocation that map to the command components. These segments represent meaningful self-contained constituents of a complex invocation. In the above instance, "under current directory" and "by REGEX" are segments representing single concepts.
We frame the problem as a translation task from English to Bash. In Section 3.1 we describe our approach to incorporate the structure in modelling
natural language invocations. Section 3.2 describes the proposed architecture and an analysis of its computational complexity at inference time.
3.1 Segmenting Invocation using Constituency Tree
The constituency tree represents the syntactic structure of a sentence based on phrase structure grammar (Chomsky, 1956). We propose a simple method that utilizes the constituency tree for segmenting natural language invocations. Our method is outlined in Figure 1. First, we normalize the invocation to replace patterns and file paths with their types. Next, we parse the normalized English invocation to obtain its constituency parse tree. For all the experiments reported in this work, we use the Stanford CoreNLP parser (Manning et al., 2014). Let the height of a node be defined as the number of edges on the longest path from the node to a leaf in the node's subtree (as shown in Figure 1). Then we perform a depth-first traversal on the tree in the left to right order of nodes. While performing the depth-first traversal, we cut the tree at the first node with a height less than a threshold and do not expand the search on this node further. As a result, we obtain various subtrees, where each subtree corresponds to a segment composed of the tokens in the leaves of the subtree. Finally, all segments are collected from the subtrees to obtain the segmented invocation.
3.2 Segmented Invocation Transformer
Let the  $nlc = [t_1, t_2, \ldots, t_n]$  be composed of  $n$  tokens. The invocation segmentation procedure takes the constituency tree for  $nlc$  and the threshold height as inputs and returns  $k$  segments  $[s_1, s_2, \ldots, s_k]$ , where each segment  $s_i = [t_j, t_{j+1}, \ldots, t_{j+n_i-1}]$  is composed of  $n_i$  tokens such that  $\sum_{i=1}^{k} n_i = n$ .
We use a Transformer (Vaswani et al., 2017) based architecture and modify the Transformer encoder to capitalize on the segmentation information obtained from the constituency tree. Specifically, an averaging layer is introduced before the Transformer encoder to capture the local structure (Section 2). From the embedded token sequence comprising of  $n$  vectors, the averaging layer computes a sequence of  $k$  segment embeddings. The input to the averaging layer consists of  $n$  vectors, each resulting from the sum of token embedding and the corresponding sinusoidal position embedding. These are grouped into  $k$  segments, and the
averaging layer then computes the mean over each segment to produce a sequence of  $k$  embedding vectors, one for each segment. On the decoder side, we use the standard Transformer decoder. We name this architecture Segmented Invocation Transformer (SIT), and it is shown in Figure 2. The model is trained by back-propagation on the cross-entropy loss with label smoothing of 0.1.
Complexity Analysis Next, we analyze the computational complexity of the cross-attention of the decoder during inference to point out the improvement over the vanilla Transformer. The decoding occurs in discrete time steps. We shall consider a single time step in this analysis. At each time step, in the cross-attention layer of the decoder, we first construct the keys, query and values and then perform a softmax over the product of keys matrix with the query vector to get the cross-attention scores. Let the dimension of the embedding vectors be  $d$ . Considering a single head for simplicity, the construction of values matrix takes  $O(kd^2)$  time (from the multiplication of  $\mathbb{R}^{k\times d}$  and  $\mathbb{R}^{d\times d}$  matrices), where  $k$  is the number of segments. Similarly, the construction of query vector takes  $O(d^2)$  time (from the multiplication of a  $\mathbb{R}^d$  vector with  $\mathbb{R}^{d\times d}$  matrix). Multiplying keys matrix  $(\mathbb{R}^{k\times d})$  with the query vector  $(\mathbb{R}^d)$  followed by a softmax (over  $k$  attention scores) takes  $O(kd + k)$  time. This step is followed by a weighted aggregation of the  $k$  values, each being  $d$ -dimensional, in  $O(kd)$  time. Hence, the overall complexity for cross-attention layer is  $O(kd^2 + d^2 + kd)$ . Since the dimension  $d$  is a constant, this can be simplified to  $O(k)$ . A vanilla Transformer would incur  $O(n)$  time. Therefore, our method provides a constant factor improvement per decoding time step. This advantage adds up due to multiple decoding time steps needed during the inference phase. The time benchmarks (Section 5.2) show these differences in practice.
4 Data and Preprocessing
For evaluating our method, we used the NL2Bash dataset (Lin et al., 2018) provided by the NLC2CMD contest (Agarwal et al., 2021) from NeurIPS 2020. It consists of approximately 10k paired English invocations and Bash commands scraped from Stack Overflow covering over 100 Bash utilities. The dataset was partitioned into five folds. We performed five runs. In each run, we split one fold equally for validation and testing. The remaining four folds were pooled to create the
Figure 2: Segmented Invocation Transformer (SIT): We introduce an averaging layer to generate embeddings for each segment. These embeddings are then fed into a standard Transformer that outputs the bash command.
training set for the run. All results mentioned in Table 3 are averaged over these five runs. Invocations and bash commands for all the models to remove file paths and regex. We used the natural language toolkit<sup>1</sup> and the Bash parser<sup>2</sup> shared by the NLC2CMD competition organizers for preprocessing.
Section 5.1 describes the experiments conducted to measure our method's accuracy. Section 5.2 describes the time benchmark. Section 5.3 describes the analysis using Integrated Gradients.
5.1 Translation Accuracy
Section 5.1.1 explains the accuracy metric proposed in the NLC2CMD competition. Section 5.1.2 lists the baselines our method is compared with. Section 5.1.3 lists the hyper-parameters and Section 5.1.4 contains a discussion of the results.
5.1.1 NLC2CMD Competition Metric
Agarwal et al. (2021) pointed out the shortcomings of existing evaluation metrics like BLEU score (Papineni et al., 2002), Exact Match accuracy and Template accuracy (Lin et al., 2018) in the context of natural language to Bash generation and proposed a new scoring mechanism for the NLC2CMD competition. This score incentivizes precision and recall of correct utility and flags weighted by the reported
1https://github.com/IBM/clai/tree/nlc2cmd/tellina-baseline/src/submission_code/nlp.tools
$^{2}$ https://github.com/IBM/clai/tree/nlc2cmd/util/bashlint
system confidence  $(\delta)$ . It ignores command arguments but considers utilities' order and flags. The translation system is also penalized for producing redundant flags.
Now, we formally describe the competition metric from Agarwal et al. (2021). Let model  $A$  output top-5 translations as follows:  $A: nlc \mapsto \{q | q = (\hat{c}, \delta)\}$ . Here the tuple  $(\hat{c}, \delta)$  represents the predicted command  $\hat{c}$  with associated confidence score  $\delta$ . We consider  $|A(nlc)| \leq 5$  and assume that there is only one ground truth command  $c$  corresponding to an invocation  $nlc$ . Then, the normalized score of a single prediction is:
$$
\begin{array}{l} S (q) = \sum_ {i \in [ 1, T ]} \frac {\delta}{T} \times \left(\mathbb {I} (U (\hat {c}) _ {i} = U (c) _ {i}) \times \right. \\ \frac {1}{2} \left(1 + \frac {1}{N _ {i}} (2 \times | F (U (\hat {c}) _ {i}) \cap F (U (c) _ {i}) | - | F (U (\hat {c}) _ {i}) \cup \right. \\ \left. F (U (c) _ {i}) |\right) - \mathbb {I} (U (\hat {c}) _ {i} \neq U (c) _ {i}) \\ \end{array}
$$
Here,  $\mathbb{I}(\cdot)$  is the indicator function,  $U(c)$  is the sequence of Bash utilities in the command  $c$ ,  $F(u)$  is the set of flags for utility  $u$  in respective command,  $T = \max \left(|U(c)|, |U(\hat{c})|\right)$  and  $N_i = \max \left(|F(U(c)_i)|, |F(U(\hat{c})_i)|\right)$ .
Total score of the prediction is defined as:
$$
S c o r e = \left\{ \begin{array}{l l} & \text {i f} S (q) > 0 \\ \max  _ {q \in A (n l c)} S (q), & \text {f o r s o m e} \\ & q \in A (n l c); \\ \frac {1}{| A (n l c) |} \sum_ {q \in A (n l c)} S (q), & \text {o t h e r w i s e}. \end{array} \right.
$$
5.1.2 Baselines
- T5 (Raffel et al., 2019): T5 is a Transformer-based model trained on large amount of data. We fine-tuned the T5-small checkpoint by huggingface (Wolf et al., 2020) on our dataset. The input to the model was "translate English to Bash:" followed by the invocation. T5-small and T5-base were tested. T5-small performed better. Results for the same are reported.
- Code-T5 (Wang et al., 2021): CodeT5 is a T5 derivative proposed to improve the performance on both code understanding and code generation tasks. It is pre-trained on eight programming languages- Java, Ruby, Javascript,
Go, PHP, Python, C and CPP. We fine-tuned the CodeT5-small checkpoint by huggingface (Wolf et al., 2020) on our dataset. The input to the model was "translate English to Bash:" followed by the invocation. CodeT5-small and CodeT5-base were tested. CodeT5-small performed better. Results for the same are reported.
- Seq2Seq (Bahdanau et al., 2015): This is an attention enhanced encoder-decoder architecture with a bidirectional LSTM encoder and a unidirectional LSTM decoder.
- Explainable-NL2BashAST (Bharadwaj and Shevade, 2021): A natural language to Bash translation model that generates explanations besides Bash commands and uses Abstract Syntax Tree information. It also uses Bash utility description besides the parallel NL2Bash data. We use the code shared by the authors at https://www.github.com/Shikhar-S/Explainable-NL-to-Bash-AST.
- Magnum (Agarwal et al., 2021): This is the winner's model from the NLC2CMD contest and the state of the art on this problem. The original system is an ensemble of multiple Transformers (Vaswani et al., 2017) trained with different seeds and batch sizes. We compare with a single model from the ensemble for fair comparison.
5.1.3 Hyper-parameters
SIT uses an embedding dimension of 256 and has 3 encoder layers with 4 attention heads each and 6 decoder layers with 8 attention heads each. It has feed-forward networks with a dimension of 1024 in both encoder and decoder layers. It is trained with a batch size of 499 tokens and gradient accumulation over 150 batches. The height threshold for cutting the subtrees, set to 4, is tuned using the performance on the validation set. Magnum takes in an embedding vector of size 512 and has 6 layers in both encoder and decoder, each with 8 attention heads. Magnum is trained for 2500 steps, with each batch containing 14000 tokens with gradient accumulation over 2 batches and a warm-up scheduler. Seq2Seq has two 256 dimensional bidirectional LSTM layers in the encoder and two 256 dimensional LSTM layers in the decoder with attention between encoder and decoder. T5-small and
<sup>3</sup>https://github.com/IBM/clai/tree/nlc2cmd/utilities/metric
Model	CPU Threads	Mean (sec)	Median (sec)	Interquartile Range (sec)
Magnum (Single Model)	1	145.70	144.51	1.98
SIT	1	59.07	59.22	1.87
Magnum (Single Model)	2	105.60	102.92	4.24
SIT	2	51.52	51.80	1.43
Magnum (Single Model)	3	94.57	92.17	2.26
SIT	3	47.63	47.62	1.82
Magnum (Single Model)	4	89.01	87.16	3.94
SIT	4	45.78	45.62	2.48
Magnum (Single Model)	5	86.73	86.01	0.72
SIT	5	46.49	46.36	2.35
Magnum (Single Model)	6	84.80	83.89	3.83
SIT	6	45.68	45.83	1.84
CodeT5-small are trained with a batch size of 32 examples, and the number of epochs is tuned based on the validation set performance. The unit of vocabulary for T5 and CodeT5 are subtokens, whereas, for all other models, the vocabulary is built from the whitespace-separated words in the NLC2CMD dataset. For all models, we use beam search with 10 beams and top-5 predictions to generate final Bash commands. We use 1 as the confidence score  $(\delta)$  for top  $p$  predictions and  $\exp (\text{beam\_score}) / 2$  for the remaining  $5 - p$ . This parameter  $p$  is tuned separately on the validation set for each model considered.
Results are reported in Table 3. SIT performs better than all other models. T5 performs the worst, probably due to the large predefined vocabulary of standard T5 models. For every other model, the vocabulary is limited to the NLC2CMD dataset. The winners of NLC2CMD also report that a decrease in vocabulary size increases performance for this task. There is no straightforward way to adapt the vocabulary of T5 and CodeT5 for this dataset. Seq2Seq performs better than T5 and CodeT5. We hypothesize that this is due to the smaller target side vocabulary learned from the data. CodeT5, trained on a large amount of code and natural language data, performs better than T5 but still lags behind SIT. Explainable-NL2BashAST performs better than T5 and Sequence to Sequence but lags behind Magnum because it is developed for commands with a single utility, and it constructs a target sequence that is twice as long as a Bash command. This makes the decoding using beam search less efficient.
Table 2: Time Benchmarks: Time taken to run inference on 1K examples from the test set. Each entry is computed from 25 repeated runs. System configuration is mentioned in Section 5.2.
Model	Test score
T5 (Raffel et al., 2019)	0.316 ± 0.021
CodeT5 (Wang et al., 2021)	0.355 ± 0.025
Seq2Seq (Bahdanau et al., 2015)	0.362 ± 0.012
Explainable-NL2BashAST (Bharadwaj and Shevade, 2021)	0.390 ± 0.012
Magnum (Single Model) (Agarwal et al., 2021)	0.428 ± 0.010
SIT (Proposed Method)	0.438 ± 0.018
Table 3: NLC2CMD Competition metric on the test set. Values range from -1 to 1 with higher being better. All entries are averaged over 5 runs and in the form mean  $\pm$  standard deviation.
Parameter Efficiency. SIT has 9M parameters, whereas Magnum has 45M parameters. This results from SIT's encoder being much smaller than Magnum's encoder, which is expected since we use the constituency parse tree information to model the natural language sequence. The 5x gain in parameter efficiency is especially important for this task since it will allow the proposed method to be employed in real-world systems with significantly less memory and power consumption.
5.2 Time Benchmark
Configuration. We consider a relatively inexpensive system configuration without access to a Graphics Processing Unit because we expect the users of our system to run it on a standard development machine. We benchmark the inference time for Magnum (Agarwal et al., 2021) and SIT (excluding constituency parsing) on the test set. The benchmarks are run on a 6 core Intel(R) Core(TM) i5-10400 CPU, using torch.utils.benchmark available in PyTorch (Paszke et al., 2019). We report results from 25 runs for each setting in Table 2. We benchmark a single model from the Magnum ensemble for
Figure 3: Alignments from Attribution Scores: Normalized invocation is on the left and normalized command is at the bottom.
comparison with a single SIT model.
Results. Results are reported in Table 2. We observe that SIT achieves a median time speedup of almost  $1.8\mathrm{x}$  over Magnum. These empirical results are in line with the complexity analysis from Section 3.2. On profiling the code, we find that a significant time spent during inference is to run the decoder, which is expected as the decoder runs in a step-wise fashion. Also, we note that the most time-consuming operations are computing self-attention and cross-attention matrices from keys, queries and values. As described in Section 3.2, SIT has fewer keys and values in the decoder cross-attention layer. Therefore, it leads to an increase in speed during inference.
5.3 Attribution Analysis
To assess if SIT attributes the probability of target tokens to correct invocation segment, we conduct an attribution analysis using Integrated Gradients (IG) (Sundararajan et al., 2017). The IG method computes attribution scores that represent each invocation segment's contribution in predicting a command token. IG takes in the trained model, a baseline invocation and an input invocation as input. The baseline invocation denotes an absence of signal to the model. We use a sequence of [PAD] tokens, corresponding to a sequence of zero embedding vectors as the baseline. Integrated Gradients are defined as the path integral of the gradients along the straightline path from the baseline to the
input. These are approximated by adding up the gradients along sufficiently small intervals on this straightline path. We used 5K steps for approximating the integral since the network is highly nonlinear.