Evaluating Inter-Bilingual Semantic Parsing for Indian Languages
Year: 2023

Abstract:
Despite significant progress in Natural Language Generation for Indian languages (IndicNLP), there is a lack of datasets around complex structured tasks such as semantic parsing. One reason for this imminent gap is the complexity of the logical form, which makes English to multilingual translation difficult. The process involves alignment of logical forms, intents and slots with translated unstructured utterance. To address this, we propose an Interbilingual Seq2seq Semantic parsing dataset IESEMPARSE for 11 distinct Indian languages. We highlight the proposed task’s practicality, and evaluate existing multilingual seq2seq models across several train-test strategies. Our experiment reveals a high correlation across performance of original multilingual semantic parsing datasets (such as mTOP, multilingual TOP and multiATIS++) and our proposed IESEMPARSE suite.

heading: 1 Introduction
text: Task-Oriented Parsing (TOP) is a Sequence to Sequence (seq2seq) Natural Language Understanding (NLU) task in which the input utterance is parsed into its logical sequential form. Refer to Figure 1 where logical form can be represented in form of a tree with intent and slots as the leaf nodes (Gupta et al., 2018; Pasupat et al., 2019). With the development of seq2seq models with self-attention (Vaswani et al., 2017), there has been an upsurge in research towards developing generation models for complex TOP tasks. Such models explore numerous training and testing strategies to further enhance performance (Sherborne and Lapata, 2022; Gupta et al., 2022). Most of the prior work focus on the English TOP settings.
However, the world is largely multilingual, hence new conversational AI systems are also expected to cater to the non-English speakers. In that regard works such as mTOP (Li et al., ∗Equal Contribution
2021), multilingual-TOP (Xia and Monti, 2021), multi-ATIS++ (Xu et al., 2020; Schuster et al., 2019), MASSIVE dataset (FitzGerald et al., 2022) have attempted to extend the semantic parsing datasets to other multilingual languages. However, the construction of such datasets is considerably harder since mere translation does not provide high-quality datasets. The logical forms must be aligned with the syntax and the way sentences are expressed in different languages, which is an intricate process.
Three possible scenarios for parsing multilingual utterances exists, as described in Figure 1. For English monolingual TOP, we parse the English utterance to it’s English logical form, where the slot values are in the English language. Seq2Seq models (Raffel et al., 2019; Lewis et al., 2020) tuned on English TOP could be utilized for English specific semantic parsing. Whereas, for multi lingual setting, a Indic multilingual TOP (e.g. Hindi Multilingual TOP in Figure 1) is used to parse Indic utterance to it’s respective Indic logical form. Here, the slot values are also Indic (c.f. Figure 1).1
The English-only models, with their limited input vocabulary, produce erroneous translations as it requires utterance translation. The multilingual models on the other side require larger multilingual vocabulary dictionaries (Liang et al., 2023; Wang et al., 2019). Although models with large vocabulary sizes can be effective, they may not perform equally well in parsing all languages, resulting in 1 In both English and Indic Multilingual TOP, the utterance and it’s corresponding logic form are in same language, English or Indic respectively.
102
overall low-quality output. Moreover, managing multilingual inputs can be challenging and often requires multiple dialogue managers, further adding complexity. Hence, we asked ourselves: "Can we combine the strengths of both approaches?"
Therefore, we explore a third distinct setting: Inter-bilingual TOP. This setting involves parsing Indic utterances and generating corresponding logical forms with English slot values (in comparison, multilingual top has non-english multilingual slot values). For a model to excel at this task, it must accurately parse and translate simultaneously. The aim of inter-bilingual semantic parsing is to anticipate the translation of non-translated logical forms into translated expressions, which presents a challenging reasoning objective. Moreover, many scenarios, such as e-commerce searches, music recommendations, and finance apps, require the use of English parsing due to the availability of search vocabulary such as product names, song titles, bond names, and company names, which are predominantly available in English. Additionally, APIs for tasks like alarm or reminder setting often require specific information in English for further processing. Therefore, it is essential to explore inter-bilingual task-oriented parsing with English slot values.
In this spirit, we establish a novel task of InterBilingual task-Oriented Parsing (Bi-lingual TOP) and develop a semantic parsing dataset suite a.k.a IE-SEMPARSE for Indic languages. The utterances are translated into eleven Indic languages while maintaining the logical structures of their English counterparts.2 We created inter-bilingual semantic parsing dataset IE-SEMPARSE Suite (IE represents Indic to English). IE-SEMPARSE suite consists of three Interbilingual semantic datasets namely IEmTOP, IE-multilingualTOP, IE-multiATIS++ by machine translating English utterances of mTOP, multilingualTOP and multiATIS++ (Li et al., 2021; Xia and Monti, 2021; Xu et al., 2020) to eleven Indian languages described in §3. In addition, §3 includes the meticulously chosen automatic and human evaluation metrics to validate the quality of the machine-translated dataset.
We conduct a comprehensive analysis of the performance of numerous multilingual seq2seq models on the proposed task in §4 with various input combinations and data enhancements. In our exper2 Like previous scenarios, the slot tags and intent operators such as METHOD_TIMER and CREATE_TIMER are respectively preserved in the corresponding English languages.
iments, we demonstrate that interbilingual parsing is more complex than English and multilingual parsing, however, modern transformer models with translation fine-tuning are capable of achieving results comparable to the former two. We also show that these results are consistent with those obtained from semantic parsing datasets containing slot values in the same languages as the utterance. Our contributions to this work are the following:
1. We proposed a novel task of Inter-Bilingual TOP with multilingual utterance (input) and English logical form (output). We introduced IE-SEMPARSE, an Inter-Bilingual TOP dataset for 11 Indo-Dravidian languages representing about 22% of speakers of the world population.
2. We explore various seq2seq models with several train-test strategies for this task. We discuss the implications of an end-to-end model compared to translation followed by parsing. We also compare how pertaining, prefinetuning and structure of a logical form affect the model performance.
The IE-SEMPARSE suite along with the scripts will be available at https://iesemparse. github.io/.

heading: 2 Why Inter Bilingual Parsing?
text: In this section, we delve deeper into the advantages of our inter-bilingual parsing approach and how it affects the dialogue management and response generation. We will address the question: “Why preserve English slot values in the logical form?”.
Limited Decoder Vocabulary: Using only English logical forms simplifies the seq2seq model decoder by reducing its vocabulary to a smaller set. This will make the training process more stable and reduce the chances of hallucination which often occurs in decoders while decoding long sequences with larger vocabulary size (Raunak et al., 2021).
Multi-lingual Models Evaluation: In this work, we explore the unique task of translating and parsing spoken utterances into logical forms. We gain valuable insights into the strengths and weaknesses of current multilingual models on this task. Specifically, we investigate how multilingual models compare to monolingual ones, how translation finetuning affects performance, and how the performance of Indic-specific and general multilingual models
Improved Parsing Latency: In figure 2, we illustrate three multilingual semantic parsing scenarios:
1. In scenario A, the Indic utterance is translated to English, parsed by an NLU module, and then a dialogue manager delivers an English response, which is translated back to Indic language.
2. In scenario B, language-specific conversational agents generate a logical form with Indic slot values, which is passed to a languagespecific dialogue manager that delivers an Indic response.
3. In scenario C, a multilingual conversation agent generates a logical form with English slot values, which is passed to an English Dialogue Manager that delivers an English response, which is translated back into Indic language.
We observe that our approach scenario C is 2x faster than A. We further discuss the latency gains and the performances differences in appendix §A. Scenario B, on the other hand, has a significant developmental overhead owing to multilingual language, as detailed below.
Handling System Redundancy: We argue that IE-SEMPARSE is a useful dataset for developing dialogue managers that can handle multiple languages without redundancy. Unlike existing datasets such as mTOP (Li et al., 2021), multilingual-TOP (Schuster et al., 2019), and multiATIS++ (Xu et al., 2020), which generate logical forms with English intent functions and slot tags but multilingual slot values, our dataset generates logical forms with English slot values as well. This
avoids the need to translate the slot values or to cre-
ate separate dialogue managers for each language,
which would introduce inefficiencies and complex-
ities in the system design. Therefore, our approach
offers a practical trade-off between optimizing the
development process and minimizing the inference latency for multilingual conversational AI agents. Finally, the utilization of a multilingual dialogue manager fails to adequately adhere to the intricate cultural nuances present in various languages (Jonson, 2002).
3 IE-SEMPARSE Creation and Validation In this section, we describe the IE-SEMPARSE creation and validation process in details.
IE-SEMPARSE Description: We create three inter-bilingual TOP datasets for eleven major Indic languages that include Assamese (‘as’), Gujarat (‘gu’), Kannada (‘kn’), Malayalam (‘ml’), Marathi (‘mr’), Odia (‘or’), Punjabi (‘pa’), Tamil (‘ta’), Telugu (‘te’), Hindi (‘hi’), and Bengali (‘bn’). Refer to the appendix §A, for additional information regarding the selection of languages, language coverage of models, and the selection of translation model. The three datasets mentioned are described below:
1. IE-mTOP: This dataset is a translated version of the multi-domain TOP-v2 dataset. English utterances were translated to Indic languages using IndicTrans (Ramesh et al., 2021), while preserving the logical forms.
2. IE-multilingualTOP: This dataset is from the multilingual TOP dataset, where utterances were translated and logical forms were decoupled using the pytext library.3
3. IE-multiATIS++: This dataset comes from the multi-ATIS++, where utterances were translated and the logical forms were generated from labelled dictionaries and decoupled, as described in appendix §3.
3 https://github.com/facebookresearch/pytext
IE-multiATIS++ Logical Form Creation The logical forms are generated from the label dictionaries, where the Intent was labeled with ‘IN:’ tag and Slots were labelled with ‘SL:’ Tags and decoupled like IE-multilingualTOP dataset. The process of generating logical forms out of intent and slot tags from the ATIS dataset is illustrated in figure 3.
IE-SEMPARSE Processing: To construct IESEMPARSE we perform extensive pre and post processing, as described below:
Pre-processing We extensively preprocess IESEMPARSE. We use Spacy NER Tagger4 to tag date-time and transform them into their corresponding lexical form. E.g. tag date time “7:30 pm on 14/2/2023." is transformed to “seven thirty pm on fourteen february of 2023."
Post-processing For many languages some words are commonly spoken and frequently. Therefore, we replace frequently spoken words in IESEMPARSE with their transliterated form, which often sounds more fluent, authentic, and informal than their translated counterparts.
To accomplish this, we replace commonly spoken words with their transliterated form to improve understanding. We created corpus-based transliteration token dictionaries by comparing Hindi mTOP, translated mTOP, and transliterated mTOP datasets. We utilize the human-translated Hindi set of mTOP dataset to filter frequently transliterated phrases and repurpose the same Hindi dictionary to postprocess the text for all other Indic languages.
4 https://spacy.io/api/entityrecognizer
3.1 IE-SEMPARSE Validation
As observed in past literature, machine translation can be an effective method to generate high quality datasets (K et al., 2021; Aggarwal et al., 2022; Agarwal et al., 2022b). However, due to inherent fallibility of the machine translation system, translations may produce incorrect utterance instances for the specified logical form. Consequently, making the task more complicated and generalizing the model more complex. Thus, it is crucial to examine the evaluation dataset quality and alleviate severe limitations accurately. Early works, including Bapna et al. (2022); Huang (1990); Moon et al. (2020a,b), has established that quality estimation is an efficacious method for assessing machine translation systems in the absence of reference data a.k.a the low-resource settings.
Using Quality Estimation: In our context, where there is a dearth of reference data for the IE-SEMPARSE translated language, we also determined the translation quality of IE-SEMPARSE using a (semi) automatic quality estimation technique. Most of recent works on quality estimation compare the results with some reference data and then prove the correlation between reference scores and referenceless quality estimation scores (Fomicheva et al., 2020; Yuan and Sharoff, 2020; Cuong and Xu, 2018). Justifying and interpreting quality estimation metrics, however, remains a stiff challenge for real-world referenceless settings.
IE-SEMPARSE Automatic Benchmarking: When a parallel corpus in both languages is
not available, it is still beneficial to benchmark the data and translation model. In our context, we conducted an evaluation of the Samanantar corpus, which stands as the most comprehensive publicly accessible parallel corpus for Indic languages (Ramesh et al., 2021). The purpose of this assessment was to emulate a scenario wherein the Samanantar corpus serves as the benchmark reference parallel dataset, allowing us to provide a rough estimate of the scores produced by quality estimation models when evaluated in a referenceless setting on a gold standard parallel translation corpus.
We use two approaches to compare English and translated text directly. For direct quality estimation of English sentences and translated sentences in a reference-less setting, we utilize Comet Score (Rei et al., 2020) and BertScore (Zhang* et al., 2020) with XLM-RoBERTa-Large (Conneau et al., 2020) backbone for direct comparison of translated and english utterances. We also calculate BT BertScore (Agrawal et al., 2022; Moon et al., 2020a; Huang, 1990), which has shown to improve high correlation with human judgement (Agrawal et al., 2022) for our three datasets and Samanantar for reference. In this case, we translate the Indic sentence back to English and compare it with the original English sentence using BertScore (Zhang* et al., 2020). The scores for the Samanantar subset on a random subset of filtered 100k phrases and our datasets IE-SEMPARSE are provided in the table 1.
Original vs Machine Translated Hindi: As the human (translated) reference was available in mTOP and multi-ATIS for Hindi language, we leveraged that data to calculate Bert and Comet score to evaluate the translation quality of our machine translation model. We notice a high correlation between both datasets’ referenceless and reference scores. Thus suggesting good translation quality for Hindi and other languages.
In table 3 comet scores and Bert scores are scores keeping original English sentence as source, original Hindi sentence as reference and translated Hindi sentence as hypothesis. For the BT BertScore, the translated Hindi sentence and the original (human-translated) Hindi sentence are back-translated (BT) back onto English and their correlation is assessed using the Bert Score.
IE-SEMPARSE Human Evaluation: In our human evaluation procedure, we employ three annotators for each language 5. We used determinantal point processes6 (Kulesza, 2012) to select a highly diversified subset of English sentences from the test set of each dataset. We select 20 sentences from IEmultiATIS++, 120 from IE-multilingualTOP and 60 from IE-mTOP. For each dataset, this amounts to more than 1% of the total test population. We then got them scored between 1-5 from 3 fluent speakers of each Indic English and Indic language by providing them with a sheet with parallel data of English sentences and subsequent translation.
Analysis. We notice that the scores vary with resource variability where languages like “as” and “kn” have the lowest scores. However, most scores are within the range of 3.5-5 suggesting the high quality of translation for our dataset. Detailed scores are reported in Appendix §B table 7.
5 every annotator was paid 5 INR for each sentence annotation each 6 https://github.com/guilgautier/DPPy

heading: 4 Experimental Evaluation
text: For our experiments, we investigated into the following five train-test strategies: 1. Indic Train: Models are both finetuned and evaluated on Indic Language. 2. English+Indic Train: Models are finetuned on English language and then Indic Language and evaluated on Indic language data. 3. Translate Test: Models are finetuned on English data and evaluated on back-translated English data. 4. Train All: Models are finetuned on the compound dataset of English + all other 11 Indic languages and evaluated on Indic test dataset. 5. Unified Finetuning: IndicBART-M2O and mBARTlarge-50-M2O models are finetuned on all three datasets for all eleven languages creating unified multi-genre (multi-domain) semantic parsing models for all 3 datasets for all languages. This can be considered as data-unified extension of 4th Setting.
Models: The models utilized can be categorized into four categories as follows: (a.) MULTILINGUAL such as mBART-large-50, mT5base such as (b.) INDIC SPECIFIC such as IndicBART (c.) TRANSLATION PREFINETUNED such as IndicBART-M2O, mBART-large-50M2O, which are pre finetuned on XX-EN translation task (d.) MONOLINGUAL (ENGLISH) such as T5-base, T5-large, BART-large, BART-base used only in Translate Test Setting. The models are specified in the table’s §8 "Hyper Parameter" column, with details in the appendix §C. Details of the fine-tuning process with hyperparameters details and the model’s vocabulary augmentation are discussed in the appendix §D and §E respectively.
Evaluation Metric: For Evaluation, we use tree labelled F1-Score for assessing the performance of our models from the original TOP paper (Gupta et al., 2018). This is preferred over an exact match because the latter can penalize the model’s performance when the slot positions are out of order. This is a common issue we observe in our outputs, given that the logical form and utterance are not in the same language. However, exact match scores are also discussed in appendix §F.5.

heading: 4.1 Analysis across Languages, Models and Datasets
text: We report the results of Train All and Unified Finetuning settings for all datasets in table 4 and 5 in the main paper as these were the best technique out of all. The scores for other train-test strategies such as translate test, Indic Train, English+Indic
Train for all 3 datasets are reported in appendix §F.1 table 9, 10 and 11 respectively. However, we have discussed the comparison between train-test settings in the subsequent paragraphs.
Across Languages: Models perform better on high-resource than medium and low-resourced languages for Train All setting. This shows that the proposed inter-bilingual seq2seq task is challenging. In addition to linguistic similarities, the model performance also relies on factors like grammar and morphology (Pires et al., 2019). For other settings such as Translate Test, Indic Train, and English+Indic, similar observations were observed.
Across Train-Test Strategies: Translate Test method works well, however end-to-end English+Indic and Train All models perform best; due to the data augmentation setting, which increases the training size.7 However, the benefits of train data enrichment are much greater in Train All scenario because of the larger volume and increased linguistic variation of the training dataset. We also discuss the comparisons in inference latency for a 2-step vs end-to-end model in §2.
Across Datasets: We observe that IEmultilingualTOP is the simplest dataset for models, followed by IE-mTOP and IE-multiATIS++. This may be because of the training dataset size, since IE-multilingualTOP is the largest of the three, followed by IE-mTOP and IE-multiATIS++. In addition, IE-multilingualTOP is derived from TOP(v1) dataset which have utterances with more simpler logical form structure (tree depth=1). IE-mTOP, on the other hand, is based on mTOP, which is a translation of TOP(v2), with more complex logical form having (tree depth>=2). We discuss the performance of models across logical form complexity in §4.2. For Unified Finetuning we observe an average performance gain of 0.2 in the tree labelled F1 score for all languages for all datasets as reported in table 5 in appendix.
Across Models: We analyse the performance across various models based on three criteria, language coverage, model size and translation finetuning, as discussed in detail below:
(a.) Language Coverage: Due to its larger size, mBART-large-50-M2O performs exceptionally well on high-resource languages, whereas IndicBART-M2O performs uniformly across all the languages due to its indic specificity. In addition, translation-optimized models perform better than
7 By 2x (English + Indic) and 12x (1 English + 11 Indic).
those that are not. mBART-large-50 outperforms mT5-base despite its higher language coverage, while mBART-large-50’s superior performance can be ascribed to its denoising pre-training objective, which enhances the model’s ability to generalize for the "intent" and "slot" detection task. In section §4.2 we discuss more about the complexity of the logical forms.
(b.) Model Size: While model size has a significant impact on the Translate Test setting for monolingual models, we find that pre-training language coverage and Translation fine-tuning are still the most critical factors. For example, despite being a smaller model, IndicBART outperforms mT5-base on average for similar reasons. Another reason for better performance for IndicBART and mBARTlarge-50 denoising based seq2seq pre-training vs multilingual multitask objective of mT5-base.
(c.) Translation Finetuning: The proposed task is a mixture of semantic parsing and translation. We also observe this empirically, when models finetuned for translation tasks perform better. This result can be attributed to fact that machine translation is the most effective strategy for aligning phrase embeddings by multilingual seq2seq models (Voita et al., 2019), as emphasized by Li et al. (2021). In addition, we observe that the models perform best in the Train All setting, indicating that data augmentation followed by fine-tuning enhances performance throughout all languages on translation fine-tuned models.
Original vs Translated Hindi: We also evaluated the performance of Hindi language models on original datasets (hiO ) and (hiIE ) which combine Hindi utterances with logical forms of English of mTOP and multi-ATIS++ datasets, as shown in ta-
ble 4. Inter-bilingual tasks pose a challenge and result in lower performance, but translation-finetuned models significantly reduce this gap. Model performance is similar for both ‘hi’ and ‘hiIE ’, indicating the quality of translations. Additional details can be refered in Appendix §G.
Domain Wise Comparison: IE-mTOP dataset contains domain classes derived from mTOP. We compare the average F1 scores for different domains in IE-mTOP dataset for IndicBART-M2O and mBART-large-50-M2O in the Train All setting, as shown in Figure 4. We observe that mBARTlarge-50-M2O outperforms IndicBART-M2O for most domains except for people and recipes, where both perform similarly well due to cultural variations in utterances.

heading: 4.2 Analysis on Logical Forms
text: In this paper, we maintain the slot values in the English language and ensure consistency in the logical form across languages for each example in every dataset. This can be useful in assessing the model performance across language and datasets on the basis of logical form structure which we have analysed in this section. Previous works have shown a correlation between model performance and logical form structures (Gupta et al., 2022).
Logical Form Complexity: We evaluate the performance of the mBART-large-50-M2O model on utterances with simple and complex logical form structures in the Train All setting for IE-mTOP and IE-multilingualTOP datasets. Simple utterances have a flat representation with a single intent, while complex utterances have multiple levels 8 of branching in the parse tree with more than one intent. In IE-multiATIS++, instances are only attributed to simple utterances since they have a single unique intent. Figure 5 shows, that mBART8 depth >= 2
large-50-M2O performs better for complex utterances in IE-mTOP, while there is better performance for simple utterances in IE-multilingualTOP due to its larger training data size and a higher proportion of simple logical forms in training data.
Effect of Frame Rareness: We compared mBART-large-50-M2O and IE-multilingualTOP on the Train All setting by removing slot values from logical forms and dividing frames into five frequency buckets9. A shown in figure 6, F1 scores increase with frame frequency, and IE-mTOP performs better for smaller frequencies while IEmultilingualTOP performs better for very large frequencies. This suggests that IE-mTOP has more complex utterances, aiding model learning with limited data, while IE-multilingualTOP’s larger training size leads to better performance in very high frequency buckets.
Post Translation of Slot Values: We translate slot values from Hindi to English using IndicTrans for the logical forms of ‘hi’ mTOP and ‘hi’ multiATIS++ datasets in the Train All setting. Table 6 compares the F1 scores of models for IE-mTOP and IE-multiATIS++ datasets, which only had the original Hindi dataset available. Despite minor decreases in scores and visible translation errors, our 9 namely very high, high, medium, low and very low.
approach yields accurate translations due to the short length of slot values and the high-resource nature of Hindi. However, we argue that our proposed task or multilingual TOP task is superior in terms of latency and performance, as discussed in §2 and §4.1.
Language Wise Correlation: We compared the logical form results of each language by calculating the average tree labelled F1 score between the datasets of one language to the other. We then plotted correlation matrices10 and analysed performance on all datasets using IndicBART-M2O and mBART-large-50-M2O in Train All setting, as described in Figure 7, 8, and 9 in Appendix §F.4.
Our analysis shows that IndicBART-M2O has more consistent predictions than mBART-large-50M2O. We also observed that models perform most consistently for the IE-multiATIS++ dataset. Additionally, related languages, such as ‘bn’ and ‘as’, ‘mr’ and ‘hi’, and ‘kn’ and ‘te’, have high correlation due to script similarity.

heading: 5 Related Work
text: Multi-Lingual Semantic Parsing: Recently, TOP has attracted a lot of attention due to the development of state-of-the-art seq2seq models such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2019). Moreover, several works have extended TOP to the multilingual setting, such as mTOP, multilingual-TOP, and multi-ATIS++. The recent MASSIVE dataset (FitzGerald et al., 2022) covers six Indic languages vs eleven in our work, and only contains a flat hierarchical structure of semantic parse. Furthermore, the logical form annotations in MASSIVE are not of a similar format to those in the standard TOP dataset.
10 for 11 x 11 pairs
IndicNLP: Some works have experimented with code-mixed Hindi-English utterances for semantic parsing tasks, such as CST5 (Agarwal et al., 2022a). In addition to these advances, there have been significant contributions to the development of indicspecific resources for natural language generation and understanding, such as IndicNLG Suite Kumar et al. (2022), IndicBART Dabre et al. (2022), and IndicGLUE Kakwani et al. (2020). Also, some studies have investigated the intra-bilingual setting for multilingual NLP tasks, such as IndicXNLI (Aggarwal et al., 2022) and EI-InfoTabs (Agarwal et al., 2022b). In contrast to prior works, we focus on the complex structured semantic parsing task.
LLMs and Zero Shot: Our work is also related to zero-shot cross-lingual (Sherborne and Lapata, 2022) and cross-domain (Liu et al., 2021) semantic parsing, which aims to parse utterances in unseen languages or domains. Moreover, recent methods use scalable techniques such as automatic translation and filling (Nicosia et al., 2021) and bootstrapping with LLMs (Awasthi et al., 2023; Rosenbaum et al., 2022; Scao, 2022) to create semantic parsing datasets without human annotation. Unlike previous methods such as Translate-Align-Project (TAP) (Brown et al., 1993) and Translate and Fill (TAF) (Nicosia et al., 2021), which generate semantic parses of translated sentences, they propose a novel approach that leverages LLMs to generate semantic parses of multilingual utterances.

heading: 6 Conclusion and Future Work
text: We present a unique inter-bilingual semantic parsing task, and publish the IE-SEMPARSE suite, which consists of 3 inter-bilingual semantic parsing datasets for 11 Indic languages. Additionally, we discuss the advantages of our proposed approach to semantic parsing over prior methods. We also analyze the impact of various models and train-test procedures on IE-SEMPARSE performance. Lastly, we examine the effects of variation in logical forms and languages on model performance and the correlation between languages.
For future work, we plan to release a SOTA model, explore zero-shot parsing (Sherborne and Lapata, 2022), enhance IE-SEMPARSE with human translation (NLLB Team et al., 2022), explore zeroshot dataset generation (Nicosia et al., 2021), leverage LLM for scalable and diverse dataset generation(Rosenbaum et al., 2022; Awasthi et al., 2023), and evaluate instruction fine-tuning models.

heading: 7 Limitations
text: One of the main limitations of our approach is the use of machine translation to create the IESEMPARSE suite. However, we showed that the overall quality of our dataset is comparable to Samanantar, a human-verified translation dataset. Furthermore, previous studies Bapna et al. (2022); Huang (1990); Moon et al. (2020a,b) have shown the effectiveness of quality estimation in referenceless settings. Lastly, we have also extensively evaluated our dataset with the help of 3 human evaluators for each language as described in §3. We can further take help of GPT4 in future to evaluate the translations in a scaled manner (Gilardi et al., 2023).
The second point of discussion focuses on the motivation for preserving logical form slot values in English. We explore the use cases where querying data in English is crucial, and how this approach can enhance models by reducing latency, limiting vocabulary size, and handling system redundancy. While open-source tools currently cannot achieve this, it would be valuable to evaluate the effectiveness of this task by comparing it with the other two discussed approaches. To accomplish this, we suggest using a dialogue manager and scoring the performance of its responses on the three TOP approaches outlined in the paper.
Another potential limitation of our dataset is that it may contain biases and flaws inherited from the original TOP datasets. However, we contend that spoken utterances are generally simpler and more universal than written ones, which mitigates the risk of cultural mismatches in IE-SEMPARSE dataset. Furthermore, our work is confined only to the Indo-Dravidian Language family of Indic languages due to our familiarity with them and the availability of high-quality resources from previous research. Nonetheless, our approach is easily extendable to other languages with effective translation models, enabling broader applications in various languages worldwide. In the future, we plan to improve our datasets by publicly releasing them through initiatives like NLLB or IndicTransV2, and by collaborating with larger organizations to have the test sets human-translated.

heading: 8 Acknowledgements
text: We express our gratitude to Nitish Gupta from Google Research India for his invaluable and insightful suggestions aimed at enhancing the quality
of our paper. Additionally, we extend our appreciation to the diligent human evaluators who diligently assessed our dataset. Divyanshu Aggarwal acknowledges all the support from Amex, AI Labs. We also thank members of the Utah NLP group for their valuable insights and suggestions at various stages of the project; and reviewers their helpful comments. Vivek Gupta acknowledges support from Bloomberg’s Data Science Ph.D. Fellowship.

heading: A Further Discussions
text: Why Indic Languages?: Indic languages are a set of Indo-Aryan languages spoken mainly in the Indian subcontinent. These languages combined are spoken by almost 22% of the total world population in monolingual, bilingual, or multilingual ways. these speakers also are the 2nd largest population of smartphone users, and almost everyone interacts with AI through chatbots. Hence it poses an excellent opportunity for NLP researchers to push state-of-the-art further for standard NLU tasks in these languages to benefit the digital business perspective and make technology more accessible to people through AI. However, most NLU benchmarks lack datasets in those languages despite some being high resource (such as ‘hi,’ ‘bn,’ and ‘pa’). Moreover, with the introduction of various NLU models like IndicBERT (Kakwani et al., 2020), indicCorp, indicBART (Kumar et al., 2022), and state-of-the-art NMT module IndicTrans (Ramesh et al., 2021) that has opened new opportunities for researchers to innovate and contribute benchmark datasets which support building NLU models for Indic languages.
Lastly, discourse in languages other than English helps society understand more diverse perspectives and leads to a more inclusive society. As the world is mainly multilingual, various studies have proven that multilingual people can contribute more diverse societal perspectives through digital discourse.
Why IndicTrans translation? Furthermore we use IndicTrans because of the following three reasons, (a.) Lightweight: IndicTrans is an extremely lightweight yet state of the art machine translation model for Indic languages. (b.) Indic Coverage: IndicTrans covers the widest variety of Indic languages as compared to other models like mBART, mT5 and google translate and azure translate are not free for research. (c.) Open Source: IndicTrans is open source and free for research purposes, more on this is elaborated in Aggarwal et al. (2022).
Why Inter-Bilingual TOP task? Task-Oriented Parsing has seen significant advances in recent years with the rise of attention models in deep learning. There have been significant extensions of this dataset in the form of mTOP (Li et al., 2021) and multilingual-TOP (Xia and Monti, 2021). However, they remain limited in terms of language coverage, only covering a few major global languages and only Hindi in the Indic category.
These datasets are especially difficult to expand to other languages due to the fact that each language has a unique word order and the logical form of each sentence should be modified accordingly. They cannot be altered using a simple dictionary lookup or alignment technique to generate a highquality dataset. In keeping with this, we propose an inter bilingual TOP task in which only input utterances are translated. As current computers continue to employ English to make decisions and interact with the outside world, modern dialogue managers can work with the logical forms of the English counterparts, construct a response, and translate it back to the input utterance’s language.
This resolves the latency issue where the model must first convert the statement to English before parsing it with another seq2seq model. This was mentioned in section §4.1 which demonstrates that end to end models perform better than translate + parsing models in certain instances. Despite the difficulties of learning translation and parsing in a single set of hyper parameters, our research demonstrates that this is feasible with existing seq2seq models, especially models that have being pretrained with translation task.
Task Oriented Parsing in the era of ChatGPT: With the rising popularity of chatGPT 11 in opendomain conversational AI. It is still a challenge to actually use these large language models in a taskoriented manner. Moreover, these open domain models may not understand the intent of the user correctly or they may take incorrect actions provided a user utterance. These LLMs also have the risk of being biased and toxic. Recent works like HuggingGPT (Shen et al., 2023) have also shown that while these models may have outstanding language understanding capabilities, it is still better to use task specific models to execute tasks in a narrow scope.
11 https://openai.com/blog/chatgpt
Model Coverages: Listed below is the language coverage for all employed multilingual models.
1. mBART-large-50: ‘bn’, ‘gu’, ‘hi’, ‘ml’, ‘mr’, ‘ta’, ‘te’
2. mT5-base: ‘bn’, ‘gu’, ‘hi’, ‘kn’, ‘ml’, ‘mr’, ‘pa’, ‘ta’, ‘te’
3. IndicBART: ‘’as‘, bn’, ‘gu’, ‘hi’, ‘kn’, ‘ml’, ‘mr’, ‘or’, ‘pa’, ‘ta’, ‘te’
4. IndicBART-M2O: ‘’as‘, bn’, ‘gu’, ‘hi’, ‘kn’, ‘ml’, ‘mr’, ‘or’, ‘pa’, ‘ta’, ‘te’
5. mBART-large-50-M2O: ‘bn’, ‘gu’, ‘hi’, ‘ml’, ‘mr’, ‘ta’, ‘te’
Two-step vs End2End parsing: We measure the translation time of IndicTrans (Ramesh et al., 2021) on an NVIDIA T4 GPU and find that it takes 0.015 seconds on average to translate a single utterance from one language to another. In scenario A, this adds 0.03 seconds of latency per utterance, while our approach only adds 0.015 seconds (≈ 12 ). In scenario B, where the logical form has slot values in Indic, there is no latency overhead for either approach, but there are significant development challenges due to multilingualism as discussed below.

heading: B Details: Human Evaluation
text: In table 7 we show the detailed scores of human evaluation process discussed in the main paper §3.

heading: C Details: Multilingual Models
text: 1. Generic Multilingual (Multilingual): these models are generic Seq2Seq multilingual models, we used mBART-large-50, mT5-base (Liu et al., 2020; Xue et al., 2021) for experiments for this category.
2. Indic Specific (Indic): These seq2seq models are specifically pretrained on Indic data, we uexplore IndicBART for experiments (Dabre et al., 2022) in this category.
3. Translation Finetuned (Translation): These pretrained seq2seq models are finetuned on the translation task with a single target language i.e. English. The models we explored form this category areIndicBART-M2O and mBART-large-50-M2O (Dabre et al., 2022; Tang et al., 2021).
4. Monolingual (Monolingual): These seq2seq models are pretrained on English data only. They were utilize only in the Translate Test setting. The models we explored form this category are T5-large, T5-base (Raffel et al., 2019) and BART-base, BART-large (Lewis et al., 2020).

heading: D Hyperparameters Details
text: In Table 8 the hyperparamaters are abbreviated as mentioned below:
1. PO: Pre-training Objective.
2. PD: Pretraining Dataset,
3. LR: Learning Rate,
4. BS: Batch Size,
5. NE: Maximum Number of Epochs,
6. WD: Weight Decay,
7. MSL: Maximum Sequence Length,
8. MS: Model Size described as a number of parameters in millions,
9. WS: Warm-up Step.
All the experiments were run on RTX A5000 GPUs in Jarvis labs 12. The code was written in PyTorch and Huggingface accelerate library 13. We used early stopping callback in training process with patience of 2 epochs for each setting.
The Average runtime for each for T5-base, BART-base, IndicBART, IndicBART-M2O was 3 minutes for IE-mTOP, 1 minute for IE-multiATIS++ and 5 minutes for IEmultilingualTOP. The Average runtime for each for T5-large, BART-large, mT5-base,mBARTlarge-50, mBART-large-50-M2O was 5 minutes for IE-mTOP, 3 minute for IE-multiATIS++ and 10 minutes for IE-multilingualTOP.
E Vocabulary Augmentation
Unique Intents and slots from each dataset (IE-mTOP, IE-multilingualTOP, IE-multiATIS++) were extracted and added to the tokenizer and model vocabulary so that the models could predict them more accurately. In a typical slot and intent tagging task, these tags would have been treated as classes in the classification model. However, since our models are trained to not predict the entire word but only subwords (Raffel et al., 2019; Lewis et al., 2020) as usually done in modern selfattention architecture (Vaswani et al., 2017), we

heading: 12 https://jarvislabs.ai/
text: 13 https://huggingface.co/docs/accelerate/index
decided to include them in the vocabulary so that they can be generated easily during prediction runtime. This also contributed to the reduction of the maximum sequence length to 64 tokens, which improved generalisation as seq2seq models generalise better on shorter sequences (Voita et al., 2021). The Excel spreadsheet containing unique slots and intents will be made accessible alongside the code and supplemental materials.

heading: F Additional Results
text: F.1 Other Train Test Settings We include the results of all other settings except Train All (Already discussed in main paper) in table 9 till 15. We have discussed the comparisons of these settings in main paper §4.1.
F.2 Translate Test vs End2End models While the performance of Monolingual models in the Translate Test setting is adequate, the performance of models in the end-to-end Train All setting outperform. Translation is prone to error, and the acquired logical form in English cannot be guaranteed to be precise. Moreover, a two-step approach to translation followed by parsing will incur greater execution time than a unified model.
F.3 Unified Models Results In unified models, we observe a gain of atleast 0.15 in all languages for all datasets for both IndicBARTM2O and mBART-large-50-M2O.
F.4 Language verses Language From figure 7, 8, 9 we observe that IndicBARTM2O is a more consistent than mBART-large-50M2O.
F.5 Exact Match Results We calculated modified exact match scores as inspired by Awasthi et al. (2023) which are agnostic of the positions of the slot tokens in the logical form. These scores are presented in tables 12, 13,
14, 15. We observed that exact match is a stricter metric as compared to tree labelled F1 (Gupta et al., 2018). We also observe that exact match scores are consistent with tree labelled F1 scores across languages, datasets and models.

heading: G Original verses Interbilingual Hindi
text: As demonstrated by figure 1, we have data accessible in Hindi for all three settings. To produce Hindi bilingual TOP data, we utilize mTOP and multi-ATIS++ to internally combine Hindi and English data tables by unique id (uid). To construct our dataset, we filter the Hindi utterances column and the English logical form columns; we refer to these datasets as hiIE in table 4. Furthermore, we conduct tests using original Hindi datasets (slot values in Hindi in logical form) and compare their performance to that of other languages. In the table 4, we refer to these datasets as hiO for the mTOP dataset and multi-ATIS++ dataset both.
Analysis. We see a decline in F1 score for all models for hiIE in both IE-mTOP and IEmultiATIS++. This might be due to data loss when hindi and english data are combined, as not all utterances of english data are included in both datasets. Furthermore, the hindi utterances in the original dataset may be more complex. The results for hiO and hiO enhances because the tokens were copied from the utterance and the model does not have to transform the tokens to English.

heading: Future Work
text: We can further take help of GPT4 in future to evaluate the translations in a scaled manner (Gilardi et al. , 2023)