Speech Recognition and Meaning Interpretation: Towards Disambiguation of Structurally Ambiguous Spoken Utterances in Indonesian
Authors: Ruhiyah Faradishi Widiaputri
Ayu Purwarianti
Dessi Puji Lestari
Kurniawati Azizah
Dipta Tanaya
Sakriani Sakti
Venue: EMNLP 2023 Main (2023)
Keywords: structural ambiguity in sentences
prosodic information
speech recognition
speech-to-text mapping
meaning interpretation
TL;DR: Indonesian ASR and meaning interpretation of structurally ambiguous utterances

Abstract: Despite being the world's fourth-most populous country, the development of spoken language technologies in Indonesia still needs improvement. Most automatic speech recognition (ASR) systems that have been developed are still limited to transcribing the exact word-by-word, which, in many cases, consists of ambiguous sentences. In fact, speakers use prosodic characteristics of speech to convey different interpretations, which, unfortunately, these systems often ignore. In this study, we attempt to resolve structurally ambiguous utterances into unambiguous texts in Indonesian using prosodic information. To the best of our knowledge, this might be the first study to address this problem in the ASR context. Our contributions include (1) collecting the Indonesian speech corpus on structurally ambiguous sentences\footnote{Our corpus is available at \url{https://github.com/ha3ci-lab/struct_amb_ind}}; (2) conducting a survey on how people disambiguate structurally ambiguous sentences presented in both text and speech forms; and (3) constructing an Indonesian ASR and meaning interpretation system by utilizing both cascade and direct approaches to map speech to text, along with two additional prosodic information signals (pause and pitch). The experimental results reveal that it is possible to disambiguate these utterances. In this study, the proposed cascade system, utilizing Mel-spectrograms concatenated with F0 and energy as input, achieved a disambiguation accuracy of 79.6\%, while the proposed direct system with the same input yielded an even more impressive disambiguation accuracy of 82.2\%.

Speech Recognition and Meaning Interpretation: Towards Disambiguation of Structurally Ambiguous Spoken Utterances in Indonesian
Ruhiyah Faradishi Widiaputri', Ayu Purwarianti', Dessi Puji Lestari',
Kurniawati Azizah², Dipta Tanaya', Sakrianí Sakti²
Bandung Institute of Technology, Indonesia
UUniversity of Indonesia, Indonesia
Japaan Advanced Institute of Science and Technology, Japan
555230140std stei. itb. ac. id, {ayu, dessipjesii ac.id, id, (kurniawati azizah, ditpa,,iiiittttee diptatanaya)@cs. ui. ac.ac.id, id, ssakti@jaist.
Abstract
Saya melihat seseorang di bukit.
EN
sow someone the hill.
Despite being the world's fourth-most popu- lous country, the development of spoken lan- guage technologies in Indonesia still needs im- provement. Most automatic speech recognition (ASR) systems that have been developed still limited to transcribing the exact word-by bby- word, which, many cases, consists of ambigu- sentences. In fact, speakers prosodic characteristics of speech to convey different interpretations, which, unfortunately, these sys- tems often ignore. In this study, we attempt to resolve structurally ambiguous utterances into unambiguous texts in Indonesian using prosodic information. To the best of our knowl- edge, this might be the first study to address this problem in the ASR context. Our contri- butions include (1) collecting the Indonesian speech corpus on structurally ambiguous sen- tences'; (2) conducting survey how people disambiguate structurally ambiguous sentences presented in both text and speech forms; and (3) constructing Indonesian ASR and meaning interpretation system by utilizing both cascade and direct approaches to map speech to text, along with two additional prosodic information signals (pause and pitch). The experimental results reveal that it is possible disambiguate these utterances. In this study, the proposed cas- cade system, utilizing Mel-spectrograms con- catenated with FO and energy input, achieved a disambiguation accuracy of 79.6%, while the proposed direct system with the same input yielded even more impressive disambigua- tion accuracy 82.2%.
Interpretation
Interpretation
Someone
Someone
Hill
Hill
Saya melihar seseorang dil bukit.
Saya mellhat seseorang dil bukit.
sow
someone
ion the hill
sew
someone
jon the hill
Figure An example a structurally ambiguous sen- tence in Indonesian and English.
every utterance contains some degree of ambigu- ity, even though alternate interpretations may not always be obvious to native speakers (Russell and Norvig, 2009). Hurford et al. (2007) categorized ambiguity into main types: lexical ambiguity and structural ambiguity. Lexical ambiguity arises when a word has multiple meanings, as seen in the word "bank," " which can refer to a financial insti- tution or the side of a river. On the other hand, structural or syntactic ambiguity occurs when a phrase sentence can be parsed in more than one way. For instance. in the sentence "I saw some- one on the hill," " the prepositional phrase "on the hill" can modify either (1) the verb "saw" (2) the noun "someone." This structural ambiguity gives rise to semantic ambiguity, resulting in different possible meanings. An example of this is illustrated in Figure 1.
1 Introduction
Efforts have been made to address the issue of ambiguity in natural language. However, the res- olution of lexical ambiguity is more commonly studied than structural ambiguity. There even specific task dedicated its resolution, namely word-sense disambiguation. Furthermore, most studies on the resolution of structural ambiguity are
Ambiguity is one of the challenges in natural lan- guage processing. It has been observed that nearly
*This work was conducted while the first author was doing internship at HA3CI Laboratory, JAIST, Japan under JST Sakura Science Program.
OOu corpus is available at https:///iithb. com/ mesi-lla/sttuct.aet

---

SPEECH
Saya mellhat seseorang. Orang itu ada bukit
Saya melihat seseorang di bukit saw someone on the hill
Disambiguation system
the hill
Figure 2: The disambiguation system.
focused on parsing tasks (Shieber, 1983; Li et al., 2014; Tran et al., 2). To the best of our knowl- edge, there have been no studies directly yielding disambiguated sentences as output in the field of structural ambiguity resolution.
nated with FO, Mel-spectrogram concatenated with energy, and Mel-spectrogram concatenated with FO and energy; (4) conducting human assessments to examine how individuals resolve structurally am- biguous sentences in written and spoken formats.
Speech is the most natural form of human-to- human communication. It is also considered the richest form of human communication in terms of bandwidth, conveying valuable supplementary information not found in text, particularly the prosodic structure (Tran et al., 2018), which lis- teners can use to resolve structural ambiguities (Price et al., 1991). Therefore, understanding spo- ken language is one of the earliest goals of natural language processing (Jurafsky and Martin, 2022). Today, automatic speech recognition (ASR) tech- nology has advanced and widely used. Unfortu- nately, most ASR systems do not typically address issues of ambiguity.
Related Work
Indonesia has one of the largest populations in the world (Cahyawijaya et al., 2021), yet the availabil- ity of resources and the progress of NLP research in Indonesian lag behind (Wilie et al., 2020; Cahyawi- jaya et al., 2022). Nevertheless, research on Indone- sian, both speech and text, is gradually progress- ing despite the challenges mentioned earlier. Even though the majority of research and existing data in Indonesian comes from the text modality, sev- eral works have attempted some speech processing tasks in Indonesian. To date, some speech corpora have also been developed (Sakti et al., 2004; Lestari et al., 2006; Sakti al., 2008a, 2013), as well the development of ASR (Sakti al., 2004, 2013; Ferdiansyah and Purwarianti, 2011; Prakoso et al., 2016; Cahyawijaya et al., 2022) and TTS (Sakti et al., 2008b; Azis et al., 2011; Mengko and Ayun- ingtyas, 2013; Azizah et al., 2020). Most Indone- sian ASRs and TTSs have achieved good results. However, specifically for ASR tasks, most ASR that has been developed in Indonesian and also in other languages is still limited transcribing the exact word-by-word, ignoring ambiguities in the transcription.
Consequently, in this work, we have made the first attempt to create system capable of disam- biguating structurally ambiguous utterances into unambiguous text by using the provided prosodic information. We propose to build the system by adapting both the cascade and direct approaches to speech-to-text mapping. The proposed cascade sys- tem combines two components: ASR and novel text disambiguation (TD) model, while the pro- posed direct system utilizes another novel speech disambiguation (SD) model. The TD model is specifically designed to transform the ASR output, which is the transcription of structurally ambiguous utterances, into the intended unambiguous text. An illustration of the disambiguation system is shown in Figure 2.
A number of attempts have also been made to address the issue of ambiguity in Indonesian, par- ticularly in the text domain. With regard to word- sense disambiguation, several studies have been conducted (Uliniansyah and Ishizaki, 2006; Faisal et al., 2018: Mahendra et al., 2018). Nevertheless. research on structural ambiguity resolution, espe- cially in speech, is far less common, and there a notable gap in this area.
Therefore, our contributions include: (1) con- structing the first Indonesian structural ambigu- ity corpus; (2) proposing both cascade and direct approaches for specch-to-text mapping to build the disambiguation system; (3) performing experi- ments by developing the disambiguation systems using the Indonesian structural ambiguity corpus created with four proposed audio input combina- tions: Mel-spectrogram, Mel-speetrogram concate-
On the other hand, some studies have attempted to utilize prosodic information in speech resolve ambiguity across various tasks. For example, Tran et al. (2018) developed a dependency parser that

---

Type Ambiguous sentences
Disambiguation sentences
The book the chair
(1) The book the chair is good. I just bought the book.
4
1 just bought is good.
(2) The book on the chair is good. 1 just bought the chair.
They buried the treasure
(1) They buried the treasure they found. They buried the park.
they found the park.
(2) They buried the treasure they found. They found it in the park.
I saw someone on the
(1) saw someone. I saw on the hill.
6
hill.
(2) saw someone. The person is on the hill.
eat spicy chicken and
(1) eat chicken and eggs. The chicken is spicy.
10
eggs.
(2) cat chicken and eggs. The chicken and eggs are both spicy.
Table 1: Examples of structurally ambiguous sentences and the results of their disambiguation for each adopted type.
incorporates acoustic-prosode features, including pauses, duration, fundamental frequency (FO), and energy. Cho et al. (2019) attempted to classify structurally ambiguous sentences into seven cate- gories of intentions using both acoustic and textual features. For the acoustic features the work com- bined root mean square energy (RMS energy) with Mel-spectrograms by concatenating them frame- wise. Tokuyama et (2021) addressed sentences with ambiguous intensity by adding an intensifier based on word emphasis. However, it's worth not- ing that most of these studies were conducted for major languages (e.g. English, Japanese, etc.). Fur- thermore, to the best of our knowledge, no stud- ies have attempted to exploit prosodic information to disambiguate structurally ambiguous utterances that directly produce unambiguous texts. There- fore, we will address this issue in our study. Al- though our focus is on the Indonesian language, the approach can potentially be applied to other languages as well.
Type verb verb + adverbial modifier. Sentences of this type become ambiguous be- cause the adverbial modifier can be considered to modify either the first or the second verb.
3. Type 6: verb noun + modifier. In this case, the modifier can be interpreted as an adverb linked to the verb or as an adjective associated with the noun.
4. Type 10: modifier + noun conjunction noun. This type of sentence becomes ambigu- ous because the modifier can be applied either the first noun or the second noun.
Through crowdsourcing, we created 100 struc- turally ambiguous sentences for each adopted type. with each ambiguous sentence having two possi- ble interpretations. These interpretations were then transformed into unambiguous texts, consisting of two sentences each. The first sentence remained the same for interpretations derived from the same ambiguous sentence, while the second sentence differed to provide clarification for the intended interpretation. As result, we obtained total of 800 pairs of ambiguous sentences and unambigu- texts. Table 1 presents examples of ambiguous sentences and their corresponding unambiguous texts for each adopted type.
3 Proposed Method
3.1 Corpus construction
Because of the limited availability of documen- tation on structural ambiguity in the Indonesian language, we turned to English-language litera- ture. Specifically, we drew upon the work of Taha (1983), who classified structural ambiguity in En- glish into twelve types. However, we chose to adopt only four of these types, focusing on those most relevant to Indonesian linguistic characteris- ties:
After validation by an Indonesian linguist, the 800 pairs of ambiguous sentences and their corre- sponding unambiguous texts were divided into 10 groups: 8 groups for training, group for valida- tion, and 1 group for the testing set. This division ensured that no sentences overlapped between the training, development, and test sets.
1. Type 4: noun noun + modifier. This type of sentence becomes ambiguous because the modifier can be considered to modify either the first noun or the second noun.
The utterances of 400 structurally ambiguous sentences were recorded in controlled, low-noise office environment. The recording involved 22 speakers, consisting of 11 males and 11 females.

---

Baseline (the common cascade approach)
ASR
saw someone on the hill
TD
saw someone. saw the hill
2. Proposed cascade approach (Cascade tag)
<0>, <t>, or <2> Added interpretation tag
ASR
saw someone on the hill
TD
saw someone. saw on the
Proposed direct approach
SD
saw someone. saw the hill
Figure Proposed approaches for building the disambiguation system.
Each speaker was assigned to different groups of ambiguous sentences ensure that neither speak- ers sentences overlapped across the training, development, and testing sets. For each pair of ambiguous sentences and their interpretations, the speaker was instructed read the ambiguous sen- tence in a way that conveyed the intended inter- pretation naturally, without overemphasis. Before the recording session, we provided the speakers with the structurally ambiguous texts they would read, allowing them to study in advance. During the recording session, closely monitored each speaker and requested corrections if their speech sounded unnatural due to overemphasis. The com- plete recording resulted in a total of 4,800 utter- ances.
biguous sentences are terminal junetures, character- ized by a pitch change followed by pause. Tran et al. (2018) mentioned four features that widely used in computational prosody models: pauses, du- ration, fundamental frequency, and energy. There- fore, in this work, we focus specifically on two prosodic cues for disambiguating structurally am- biguous utterances: pauses and pitch variation. addition to these features, we also extracted Mel- spectrogram for speech recognition.
The features extracted for pause in this work were Mel-spectrogram and energy. These features were chosen because they indirectly reflect the pres- ence of pauses. In a low-noise enviromment, when pause occurs, all channels in the Mel-spectrogram and energy exhibit values close to zero. To repre- sent pitch, we extracted FO. These features were then concatenated with the Mel-spectrogram frame- wise, following a similar approach described by Cho et al. (2019). Therefore, in this work. four input combinations were proposed: (1) Mel- spectrogram only; (2) Mel-spectroogrm concate- nated with FO; (3) Mel-spectrogram concatenated with energy (root mean square energy RMS en- ergy); and (4) Mel-spectrogram concatenated with FO and energy (RMS energy).
3.2 Disambiguation of structurally ambiguous sentences in Indonesian
To develop system capable of disambiguat- ing structurally ambiguous utterances based on prosodic information, two key considerations arise: determining the necessary prosodic features for dis- ambiguation and developing the disambiguation system.
3.2.1 Utilization of prosodic information for disambiguation
3.2.2 Development of disambiguation system The disambiguation system functions as speech- to-text mapping, taking structurally ambiguous ut- terances as input and producing unambiguous text as output. Similar to other speech-to-tess map- ping tasks, the disambiguation system be im- plemented using two approaches: the cascade ap- proach, which combines two or more components, and the direct approach, which transforms struc- turally ambiguous utterances into unambiguous
According to Taha (1983), several prosodic sig- nals that can be used to disambiguate structurally ambiguous sentences include emphasis, pitch, and juncture. Emphasis encompasses changes in pitch, duration, and energy during speech, as noted by Tokuyama et al. (2021). The term 'juncture' refers speech features that allow listeners to detect word or phrase boundaries. Specifically, junctures that can be used to disambiguate structurally am-

---

text without any intermediate representation. In this study, as we described earlier, we propose the cascade disambiguation system, which consists of two components: the automatic speech recognition (ASR) model and a novel model known as the text disambiguation (TD) model. Additionally, we in- troduce the direct disambiguation system, featuring another novel model called the speech disambigua- tion (SD) model.
the second noun.
2. Type 5: verb + verb + adverbial modifier:
tag < 1 > was given if the modifier was inter- preted as modifying the first verb; tag 2 was given if the modifier was interpreted modifying the second verb.
3. Type 6: verb + noun modifier: tag given if the modifier was interpreted modifying the verb; tag > was given if the modifier was interpreted as modifying the noun.
is evident that the disambiguation system ap- proaches bear a resemblance to the approaches used in speech-to-tex: translation, which involves the automatic translation of utterances from one lan-
guage to texts in another language (Bentivogli et al.,
4. Type 10: modifier + noun conjunction noun: tag > was given if the modifier was interpreted as modifying only one noun; tag < 2 > was given if the modifier was inter- preted as modifying both nouns.
2021). The key difference lies in the replacement of the machine translation model (MT) with the text disanbiguation model (TD). While the MT model was responsible for translating the transcrip- tion of ASR output, the TD model was specifically designed to convert the ASR output, consisting of structurally ambiguous sentences, into unambigu- ous text in the same language.
For comparison, we also developed the disam- biguation system using the original cascade ap- proach (with no tag addition) as a baseline. Regard- ing the proposed direct approach, which inherently preserves all the necessary information for gener- ating unambiguous text, it required significant modifications from the standard direct approach. Hence, the proposed architectures are illustrated in Figure 3.
'The proposed cascade approach may encounter challenges when standard ASR is applied without modification, leading the loss of crucial prosodic information needed for disambiguating structurally ambiguous utterances. This prosodic information loss prevents TD from generating the desired un- ambiguous texts. Hence, to build the disambigua- tion system, modifications must be made so that the TD knows not only the structurally ambigu- ous transcription but also which interpretation is intended by the utterance based the prosodic information given. In this work, we propose to do this by adding a special 'meaning tag' indicating the intended interpretation based on the utterance's s prosodic information. Thus, in the proposed cas- cade approach, the ASR generates not only the transcription of structurally ambiguous utterances but also the special tags. Since each structurally ambiguous sentence in this work was limited to having exactly two interpretations, we proposed add a 'meaning tag" for each sentence: < 0 > for unambiguous sentences and < > or <>> for structurally ambiguous sentences. The tagging as '1' or '2' must be uniform for each structurally ambiguous sentence of the same type:
To effectively train the ASR and SD models, re- lying solely on the structural ambiguity corpus we created is insufficient. These models require more data to recognize speech accurately. Therefore, in addition to using the structural ambiguity corpus. incorporated another speech corpus for training both ASR and SD. The structural ambiguity cor- pus we created was also insufficient for training the TD model. To overcome this limitation, we employed the transfer learning technique, utiliz- ing a pre-trained model for the TD. Specifically, we used BART (Lewis et al., 2020), well- well-known pre-training model suitable for TD tasks due to its sequence-to-ssquueee architecture.
4
Experiment
4.1 Human assessment
We conducted survey involving 20 participants to investigate how people disambiguate structural ambiguity in Indonesian. This investigation encom- passed two conditions: one in which the sentences were presented as text only and another in which they were presented as speech. To evaluate how individuals disambiguate structurally ambiguous
1. Type 4: noun + noun modifier: tag < 1 was given if the modifier was interpreted as modifying the first noun; tag < was given if the modifier was interpreted as modifying

---

Indo-
Type
Type
Type
Type
Total
LVCSR4
5
6
10
0.8
Train
41500
1040
1040
1040
1040
45660
0.0
Dev
160
80
80
80
80
480
Test
160
80
80
80
80
480
0.4
Table 2: Data used for training, developpeet, and testing of ASR and SD models.
Type 5
Type
Types
LVCSR
Struct. amb
Figure 4: How unambiguous the sentences per type humans in text and speech.
(tag = 0)
tag 1
tag =
Train
320
80 x types
80 x 4 types
Dev
40
10 x 4 types
10 4 types
Test
40
10 x 4 types
10 4 types
sentences in text form, each participant was pre- sented with 40 ambiguous sentences and asked to select the most probable interpretation for each one. The disambiguation result for text-only sentences was determined by the interpretation chosen most frequently. To evaluate how people disambiguate structurally ambiguous sentences in speech form, each participant was asked to listen to 80 struc- turally ambiguous utterances and choose the most appropriate interpretation based on the recording. Participants were also given the opportunity add their own interpretations they believed they could provide more precise representation than the two given options.
Table 3: Data used for training, development, and testing of TD models.
tags, as mentioned earlier. The SD model also ac- cepted audio input but produced unambiguous text as its output.
As mentioned previously, our training for both the ASR and SD models involved not only the structural ambiguity corpus we created but also an additional speech corpus. Specifically, we uti- lized the Indonesian LVCSR news corpus (Sakti et al., 2008a, 2004, 2013) This corpus consists of 400 speakers, each reading 110 sentences, resulting in more than 40 hours of speech. The sentences in the Indonesian LVCSR were unambiguous, the additional meaning tag for the sentences was < 0 >, while the target output for the SD model for these sentences was the sentences themselves with- out any modifications. To ensure that there were no speakers or sentences that overlapped in the training, development, and test sets, we selected a subset of the LVCSR data. The specifics regarding the amount of data used for training, development, and testing of both the ASR and SD models can be found in Table 2.
Figure 4 illustrates the degrec of ambiguity per- ceived by the survey participants for each sentence type, both in text form and speech. According the survey results, structurally ambiguous sen- tences are difficult to disambiguate for people when presented text forn. The limited information conveyed solely through text often leads to sub- jective interpretations that vary among individuals. On the other hand, the majority of structurally am- biguous utterances can be correctly disambiguated by people when presented in speech. This suggests the presence of additional information in speech that may not be available in text alone, aiding indi- viduals in disambiguating structurally ambiguous sentences. One source of this additional informa- tion is prosody.
In this work, we constructed the TD system by fine-tuning the IndoBART model (Cahyawijaya et al., 2021)². The TD took the meaning tag out- putted by the ASR concatenated with the ASR tran- scription, which is the structurally ambiguous sen- tence, as input and then outputted the unambiguous sentence.
4.2
Machine Evaluation
4.2.1 Experimental set-up
All models utilized in this study were based on the Transformer architecture (Vaswani et al., 2017). We utilized the Dpeech-Transforme architecture (Dong al., 2018) for both the ASR and SD mod- cls. The ASR component received audio input and generated transcriptions along with the meaning
The data used for the TD model consisted of 800 pairs of previously constructed structurally am-
"This corpus is publicly available at https://github. com/s- www.s/aataaaiinneeeenaa attssssatiiddataiddd_eeeeeeee insptt__iid___ee____ee httpps: / /gg thub. com/IndoNLP/indonng

---

CER
WER
Tag acc.
Baseline
TD tag
Mel
BLEU
65.6
73.5
Baseline
0.0621
0.2358
ROUGE-1
91.7
96.3
ASR + tag
0.0609
0.2140
0.9729
ROUGE-2
83.8
91.4
Mel + FO
ROUGE-L
91.7
96.2
Baseline
0.0574
0.2157
Based
on
0.65
0.875
ASR + tag
0.0626
0.2294
0.975
Disam-
WER scores
Mel RMS energy
biguati-
Based
0.658
0.908
Baseline
0.0617
0.2247
acc.
BLEU
ASR tag
0.0691
0.2375
0.9708
scores
Mel FO RMS energy
Based on ROUGE-1 0.642
0.917
Baseline
0.0638
0.2279
ASR + tag
0.0575
0.2061
0.9875
scores
Based on ROUGE-2 0.642
0.883
Table 4: Comparison of CER, WER, and tag accuracy of the proposed ASR model with the baseline using four different input combinations.
scores
Based
0.642
0.925
ROUGE-L
biguous sentences, along with 400 unambiguous sentences taken from the transcription of the In- donesian LVCSR corpus. The development and test sets in TD shared the same sentences as ASR and SD, while the training set in TD only included subset of sentences from the ASR and SD train- ing data ensure a balance of the meaning tags in the TD training data, Details about the amount of data used for TD training, validation, and testing can be found in Table
scores
Table Comparison of BLEU, ROUGE, and disam- biguation accuracy of the proposed TD model with the baseline.
input Mel-spectrograms concatenated with FO and RMS energy achieved slightly better WER and tag accuracy compared other ASRs.
In this work, the evaluation of the TD models in- volved three types of metrics: BLEU, ROUGE, and the accuracy of the disambiguation results. BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) were selected due to the similarity of the TD task in this work to paraphrasing. The disambiguation results in this work were determined by classifying the unambiguous texts generated by the TD models based on their most suitable interpretation. The interpretation of an unambiguous text was deter- mined using WER, BLEU, ROUGE-1, ROUGE- 2, and ROUGE-L scores in comparison to three other texts: the non-disambiguated text, the disam- biguated text with interpretation < >, and the disambiguated text with interpretation < 2 This classification method based on the intuition that the true TD output will exhibit the highest sentence similarity with the intended unambiguous text. Table 5 presents the evaluation results of two TD models: the bascline TD with no tag addition and the TD with additional tags. The table clearly illustrates significant improvement in TD per- formance achieved through the addition of tags, particularly in terms of disambiguation accuracy, which reaches approximately 90%. This under-
The hyperparameter configurations used in this study are provided in detail in Appendix A.
4.2.2 Experiment result
Table displays the evaluation results for base- line ASR models and ASRs augmented with tags in four input combinations: (1) Mel-spectrogram; (2) Mel-spectrogram concatenated with FO; (3) Mel-spectrogram concatenated with RMS energy: and (4) Mel-spectrogram concatenated with FO and RMS energy. The transcription produced by the ASRs was evaluated using CER (character error rate) and WER (word error rate), while the tags generated by the ASRs were assessed for accu- racy. As shown in the table, most ASRs exhib- ited similar CERs and WERs, approximately 0.06 and 0.22, respectively, indicating satisfactory tran- scription performance. Similarly, the tagging ac- curacy showed consistent trend, with accuracy surpassing 0.9, even for input Mel-spectrograms. This suggests that pauses, already captured in the Mel-spectrograms, already provide valuable infor- mation for disambiguating structurally ambiguous sentences. However, ASR with additional tags and

---

Disambiguation accuracy
BLEU
ROUGE- ROUGE- ROUGE-
Based on WER
Based on BLEU scores
Based
Based on ROUGE-
Based on ROUGE- I. scores
L
on ROUGE-
2 scores
Scores
Mel
Cascade baseline
33.7
71.0
52.0
70.3
0.571
0.56
0.569
0.558
0.556
Cascade tag
42.8
76.1
59.0
75.9
0.785
0.763
0.815
0.777
0.817
Direct
41.3
74.8
57.2
74.2
0,802
0.783
0.844
0.806
0.842
Mel + FO
Cascade baseline
36.9
72.7
53.5
72.2
0.588
0.575
0.573
0.575
0.573
Cascade + tag
40.8
74.8
56.7
74.7
0.773
0.733
0.817
0.773
0.815
Direct
39.6
74.3
56.1
73.4
0.788
0.765
0.829
0.769
0.802
Mel RMS energy
Cascade baseline
35.2
71.7
53.1
71.2
0.59
0.592
0.583
0.585
0.579
Cascade tag 39.4
73.7
55.7
73.5
0.781
0.742
0.817
0.771
0.815
Direct
41.2
74.4
56.6
73.5
0.821
0.765
0.84
0.8
0.827
Mel + FO + RMS energy
Cascade baseline
34.7
71.7
51.8
71.4
0.61
0.588
0.598
0.588
0.598
Cascade tag
41.7
76.8
60.0
76.6
0.798
0.763
0.827
0.798
0.827
Direct
42.0
75.8
58.4
75.0
0.825
0.769
0.856
0.817
0.844
Table 6: Comparison of BLEUU, ROUGE, and the disambiguation accuracy between the proposed cascade and direct models.
scores the crucial role of prosodic information, pre- served by the tags, in enabling the TD model to generate the intended unambiguous text. The TD model achieves commendable scores, especially in ROUGE-I, ROUGE-2, and ROUGE-L, high- lighting the positive impact of the transfer learning method. These high scores indicate a close align- ment between the generated disambiguation text and the desired unambiguous text.
lenging task, is trained using the same model and data as ASR without any transfer learning methods, while in the cascade approach, the task is divided between ASR and TD, and the TD models were pre-trained beforehand. However, the difference between both systems is not significant. This is because the output text from the disambiguated results is not considerably different from the am- biguous transcriptions, making it not very hard to learn. On the other hand, in terms of the accu- racy of the disambiguation results, the direct sys- tems demonstrate better performance, showing a difference of approximately 2.5%. This due the error propagation within cascade system. The errors generated by the ASR, including tran- scription and tagging errors, result in decrease in disambiguation accuracy of approximately 10% compared to the scenario where TD receives error- free transcription and tagging, shown in Table 5.
The cascade and direct systems were evaluated with the same metrics as TD: BLEU, ROUGE, and disambiguation accuracy. Table shows the com- parison between the cascade and direct systems built with four combinations of input. As can be seen in the table, both cascade and direct systems with the best performance use mel spectrograms combined with FO and RMS energy input, in the case of ASR. Then, similar to the case of TD, the cascade systems with additional tags exhibit significantly better performance than the baseline cascade systems that do not include any additional tags.
In summary, in terms of average BLEU, ROUGE-1, ROUGE-2, and ROUGE-L scores, the proposed cascade system with Mel-spectrograms concatenated with FO and energy (RMS energy) as input achieves a slightly higher score of 63.8. On the other hand, the proposed direct approach system with Mel-spectrograms concatenated with
As can be seen in Table 6, cascade mod- els slightly outperform direct systems in terms of BLEU, ROUGE-1, ROUGE-2, and ROUGE- L scores. This makes sense considering that in the cascade approach, SD, which has a more chal-

---

FO and RMS energy as input significantly improves disambiguation accuracy to 82.2%. Since our pri- mary focus in this study disambiguation, we can conclude that the direct system is the best in this case.
have more than two interpretations and enabling the speech recognizer to handle disfluencies. Addi- tionally, we will further enhance the performance of ASR, TD, and SD models to improve the disam- biguation of structurally ambiguous sentences.
5 Conclusions
Ethics Statement
This paper introduces the first Indonesian structural ambiguity corpus, comprising 400 structurally am- biguous sentences, each paired with two unambigu- ous texts, totaling 4800 speech utterances. We also conducted an evaluation of human performance in disambiguating written and spoken structurally ambiguous sentences, highlighting the advantages of the spoken format. Additionally, we developed systems for disambiguating structurally ambiguous sentences in Indonesian by adapting both cascade and direct approaches to opeech-to-text mapping, using two additional prosodic information signals (pauses and pitch). The experiment demonstrates that by utilizing prosodic information, structurally ambiguous sentences can be disambiguated into unambiguous interpretations. Among the systems tested, the proposed cascade system achieved aver- age BLEU, ROUGE-1, ROUGE-2, and ROUGE- L scores of 63.8 and disambiguation accuracy of 79.6%. In contrast, the proposed direct sys- tem, which uses input Mel-spectrograms concate- nated with FO and energy (RMS energy), obtained slightly lower average BLEU, ROUGE-I, ROUGE- 2, and ROUGE-L scores of 62.8, but it achieved the highest disambiguation accuracy of 82.2%. This represents a significant improvement compared to the proposed cascade system.
This research study has been reviewed and ap- proved by the Institutional Review Board of the Research Ethics Committee of the Japan Advanced Institute of Science and Technology (JAIST) under the research project titled "Construction of Mul- tilingual Al Speech Language Technology That Imitates Human Communication to Overcome Lan- guage Barriers," in collaboration with Indonesian institutions (approval number 04-403, approval date: January 26th, 2023). Based on this approval, the experiments were conducted in accordance with the institutional ethics guidelines.
First, all annotators, speakers, and respondents in this study were recruited through public announce- ments. The selection process for crowdsourced participants followed a 'first in, first out' (FIFO) basis while ensuring that they met specific crite- ria: being native speakers of Indonesian, university students, and aged over 20 years old, These crite- ria were chosen based on the belief that university students who are native speakers would already possess strong Indonesian language skills.
Second, all participants were provided with in- formation about research objectives, data usage and the protection of their personal information. We ensured that all annotators, speakers, and re- spondents involved in this study were fully aware that their data would be used exclusively for re- search purposes and would not be misused for other reasons. We removed any information that could identify individuals, such as speaker names, and anonymized them immediately after the experiment (e.g., speaker F01). Names were used only tem- porarily and not stored. It's important to note that while the speech data itself could potentially serve as a personal identification code, for this research, we needed to retain the speech data in its original form due to its nature.
Limitations
In this work, the structurally ambiguous sentences created and handled by the disambiguation systems were adaptations of types 4, 5, 6, and 10 in the types of structurally ambiguous sentences in En- glish by Taha (1983), adapted to Indonesian. Each structurally ambiguous sentence in this work was limited to having only two interpretations. Conse- quently, the sentences were relatively simple, with a maximum word count of 15. Furthermore, the ut- terances did not contain any disfluencies and were recorded in low-noise environment. However, this study presents a potential solution for disam- biguation. Therefore, further improvements will aim to address the current system limitations, such as allowing structurally ambiguous sentences to
Third, those participants who agreed provided written consent participate in this study, in ac- cordance with the institutional ethics guidelines. After the experiments, they received appropriate compensation based on the hourly work policy at our institution.

---

Acknowledgements
svm. In 2018 Interational Seminar on Application for Technology of Information and Communication, pages 243. IEEE.
Part of this work is supported by JSPS KAKENHI Grant Numbers JP21H05054 and JP2160 as well as JST Sakura Science Program.
Veri Ferdiansyah and Ayu Purwarianti. 2011. In- donesian automatic speech recognition system us- ing english-based acoustic model. In Proceedings of the 2011 International Conference on Electrical Engineering and Informatics, pages 1-4, IEEE
References
Nur Aziza Azis, Rose Maulidiyatul Hikmah, Teresa Va- nia Tjahja, and Anto Satriyo Nugroho. 2011. Eval- uation of text-to-speech synthesizer for indonesian language using semantically unpredictable sentences test: indotts, espeak, and google translate tts. In 2011 International Conference Advanced Computer Sci- ence and Information Systems, pages 237-242. IEEE
James Hurford, Brendan Heasley, and Michael B Smith. 2007. Semantics: a coursebook. Cambridge university press.
Daniel Jurafsky and James H Martin. 2022. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (3rd (draft) ed.).
Kurniawati Azizah, Mirna Adriani, and Wisnu Jatmiko. 2020. Hierarchical transfer learning for multilingual, multi-speaker, and style transfer dan-based tts on low-resource languages. IEEE Access, 8:179798 179812.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 0rXiv:1412.660
Dessi Puji Lestari, Koji Iwano, and Sadaoki Furui. 2006. A large vocabulary continuous speech recognition system for indonesian language. In 15th Indonesian Scientific Conference in Japan Proceedings, pages 17-22.
Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus direct speech translation: Do the differences still make a differ- ence? In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- val Language Processing (Volume 1: Long Papers). pages 2887,2887, Online. Association for Computa- tional Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 787 1871-7880, Online. Association for Computa- tional Linguistics.
pre-training
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Indra Winata, Bryan Wilie. Rahmad Mahen- dra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Fajri Koto, et al. 2022. Nusacrowd: Open source initiative for indonesian nlp resources. arXiv preprint arXiv:2212.09645
Zhenghua Li, Min Zhang, and Wenliang Chen. 2014. Ambiguity-aware ensemble training for semi- supervised dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 457-467, Baltimore, Maryland. Association for Computational Linguistics.
Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kun- coro, Sebastian Ruder, Zhi Yuan Lim, Syafri Ba- har, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNIGG: Benchmark and resources for evaluating Indonesian natural language generation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 8875-8898, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Chin-Yes Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Rahmad Mahendra, Heninggar Septiantri, Haryo Ak- barianto Wihowo, Ruli Manurung, and Mirna Adri- ani. 2018. Cross-lingual and supervised learning approach for Indonesian word sense disambiguation task. In Proceedings of the 9th Global Wordnei Con- ference, pages 245-250, Nanyang Technological Uni- versity (NTU), Singapore. Global Wordnet Associa- tion.
Won IK Cho. Jeonghwa Cho. Woo Hyun Kang, and Nam Soo Kim. 2019. Text matters but speech influ- ences: A computational analysis of syntactic ambi- guity resolution. arXiv preprint 190.09.5
Linhao Dong. Shuang Xu, and Bo Xu. 2018. Speech- transformer: a no-recurrence seguence-to-sequennc model for speech recognition. In 2018 IEEE inter- national conference on acousties. speech and signal processing (ICASSP), pages 5884 5888. IEEE
Richard Mengko and Aulia Ayuningtyas. 2013. Indone- sian text-to- toweeeccsehhheeooe system using syllable concatena- tion: Speech optimization. In 2013 3rd International Conference Instrumentation, Communications, In- formatiio Technology and Biomedical Engineering (ICICI-BME), pages 412-415. IEEE
Edi Faisal, Farza Nurifan, and Riyanarto Samo. 2018. Word sense disambiguation in bahasa indonesia using

---

Sashi Novitasari, Sakriani Sakti, and Satoshi Nakamura. 2022. A machine speech chain approach for dynami- cally adaptive lombard Its in static and dynamic noise environments. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2673-2688.
Hirotaka Tokuyama. Sakriani Sakti, Katsuhito Sudoh, and Satoshi Nakamura. 2021. Transcribing par- alinguistic acoustic cues to target language text in transformer-based speech-to-text translation. In In- terspeech, pages 2262-2266.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- kational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Trang Tran, Shubham Toshniwal, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Mari Ostendorf. 2018. Parsing speech: a neural approach to integrating lexi- cal and acoustic- acoustic-prosodic information. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume I (Long Papers), pages 69 81, New Orleans, Louisiana. Association for Computational Linguisties.
Hamdan Prakoso, Ridi Ferdiana, and Rudy Hartamto. 2016. Indonesian automatic speech recognition sys- tem using cmusphinx toolkit and limited dataset. In 2016 International Sympoxium on Electronics and Smart Devices (ISESD), pages 283-286, IEEE.
Mohammad Teduh Uliniansyah and Shun Ishizaki. 2006. A word sense disambiguation system using modified naive bayesian algorithms for indonesian language. Information and Media Technologies, 1(1):257-274.
Patti Price, Mari Ostendorf, Stefamie Chattuck-Hufnagee and Cynthia Fong. 1991. The use of prosody in syn- tactic disambiguation.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Speech and Natural Lan- guage: Proceedings of a Workshop Held at Pacific Grove, California, February 19- 1991.
Stuart J. Russell and Peter Norvig. 2009. Artificial Intelligence: a modern approach, 3 edition. Pearson.
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Compuiational Linguistics and the 10th International Joint Conference Natural Lan- guage Processing, pages 843 857, Suzhou, China. Association for Computational Linguistics.
Sakriani Sakti, Paulus Hutagaol, Arry Akhmad Arman, and Satoshi Nakamura. 2004. Indonesian speech recognition for hearing and speaking impaired people. In Proc. International Conference on Spoken Lan- guage Processing (INTERSPEECH-.. SIRPRPEECCHICSLP), pages 1037-1040, Jeju Island, Korea.
Sakriani Sakti, Eka Kelana, Hammam Riza, Shinsuke Sakai, Konstantin Markov, and Satoshi Nakamura. 2008a. Development of Indonesian large vocabu- lary continuous speech recognition system within s-SR project. In Proceedings of the Workshop on Technologies and Corpora for Asia-Pacific Speech Translation (TCAST).
A Hyperparameter Setting
The ASR and SD model training in this study did not involve hyperparameter tuning since they al- ready employed almost identical hyperparameters to those used in the original Speech-transfore (Dong et al., 2018). The ASR and SD models were based the trancoerransformmer, which consisted of 12 encoder blocks, 6 decoder blocks, 4 heads, and a
Sakriani Sakti, Ranniery Maia, Shinsuke Sakai, Tohru Shimizu, and Satoshi Nakamura. 2008b. Develop- ment hmm-base based indonesian speech synthesis. In Proc. Oriental COCOSDA, volume 1.
Sakriani Sakti, Michael Paul, Andrew Finch, Shinsuke Sakai, Tat Vu Thang, Noriyuki Kimura, Chiori Hori, Bicchiro Sumita, Satoshi Nakamura, Jun Park, Chai Wutiwiwatchai, Bo Xu, Hammam Riza, Karunesh Arora, Chi Mai Luong, and Haizhou Li. 2013. A- STAR: Toward translating asian spoken languages. Special issue on Sreechh-t--Speeee Translation, Com- puler Speech and Language Journal, 27(2):09-5527.
inner dimension of 2048. We used 80 dimensions for the Mel-spectrogram in- put. The models were trained using the Adam opti- mizer (Kingma and Ba, 2014) with 31 = 0.9, B2 = 0.98, - 10-9 and employed cross-entropy loss with neighborhood smoothing. The learning rate varied during training with warm-up period. De- coding was performed using beam search with a beam size of 10. The Opeech-transforme model employed this study is based on the ASR source code from the research conducted by Novitasari et al. (2022).
Stuart M. Shieber. 1983. Sentence disambiguation by shift-reduce parsing technique. In 21st Annual Meet- ing of the Association for Computational Linguistics, pages 113-118, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Abdul Karim Taha. 1983. Types of syntactic ambiguity in english.

---

Learning rate
Max epoch
Baseline
10
10
Cascade tag
10-4
50
Table 7: Hyperparameter tuning results for each TD model.
For TD, we aligned most hyperparameters with the optimal settings recommended for fine-tuning IndoBART, as provided by the IndoNLG paper (Cahyawijaya et al., 2021). These settings included a batch size of 8, early stopping after no improve- ments in batches, step decay learning rate scheduler with step of I epoch and gamma of 0.9, and fine-tuning with the Adam optimizer (31 0.9, 32 0.999, e 10-5). However, also conducted simple hyperparameter tuning for the learning rate and maximum epochs within each tagging method (baseline and cascade+tag). We explored three combinations learning rates and maximum epochs (learning rate=10-4 and max- imum epochs=10, learning rate=10 4 and max- imum 50 learning ratc=10-5 and maxi- epochs=10), while keeping the other hyper- parameters consistent with the settings specified in the IndoNLG paper. The results of the hyperparam- eter tuning are presented in Table 7.


Reviews:
[{"review_id": "KqTVzOrQ3G", "reviewer": "Reviewer_S9Q7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for the insightful and constructive review. Below we address the reviewer's concerns point by point. \n1. **Weaknesses are that the created corpus is quite small, not sure what differences may be significant**  \n   Answer: One of the biggest problems encountered in research on the Indonesian language is the limited amount of data available, especially speech data. However, here we managed to show that even with small data there is a possibility for building an Indonesian speech disambiguation system.\n   \n2. **Could you consider initializing the ASR decoder with IndoBART as an additional comparison to account for the additional training data the cascade model may have seen?**   \nAnswer: Our focus in this research is to compare cascade and direct approaches of disambiguation systems with their ASR and SD using the same model. However, we will keep your suggestion in mind for future research possibilities.\n\nWe also appreciate your input on typos, grammar, style, and presentation improvements. If this paper is accepted, we will make the necessary enhancements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough treatment of a new area of study for Indonesian, including corpus creation, a human assessment, and experimental comparison ", "reasons_to_reject": "- Weaknesses are that the created corpus is quite small, not sure what differences may be significant; ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses structural ambiguity in Indonesian ASR through three key contributions:  \n(1) collecting an Indonesian speech corpus with 4800 utterances (800 pairs of ambiguous sentences covering four types of structural ambiguity and spoken by multiple speakers);  \n(2) human assessment study on how well speakers are able disambiguate structurally ambiguous sentences in both speech and text, giving context and an upper bound for later experiments;  \n(3) training ASR models and studying the impact of prosodic features (pitch and pause) as well as decoupling the ASR task from structural disambiguation (by either using a single direct model, or a cascade of ASR plus features/tagging and re-generation with a finetuned IndoBART model to generate unambiguous sentence(s). Experiments show that prosodic information is (expectedly) crucial for this task. \n\nThe paper is clearly written, with illustrative examples, and takes a thorough approach to the topic, from corpus creation, to speaker study, to model training and methodology. Some small table formatting cleanup would be helpful.  \nThe inclusion of the human study is quite nice for context, showing the expected upper bound on text-only disambiguation and speech for downstream models, and that it differs slightly by type.  \nThe created corpus is quite small, and read speech with subjects knowing the task which may lead to exaggerated or less natural prosody, but, would be challenging to collect otherwise and newly enables study of this particular phenomenon in Indonesian. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The metrics (BLEU, ROUGE, WER, accuracy) in Tables 4-6 are typically reported in the 0-100 range; some scores appear to be formatted this way and some 0-1, please standardize (to 0-100). \n\nTable 6: Please report only 1 decimal place for BLEU, ROUGE, or WER. The metrics are not sensitive enough to require 3 decimal places and it will make the table easier to read. Consider using \\resizebox{} so ROUGE will not be broken onto 2 lines. Why is ROUGE-1 in the second to last columns in Table 6?  Perhaps state in the table caption what is being highlighted with red text\n\nAdditional training details to aid reproduction of experiments should be added at least to the appendix, for example number of finetuning steps, batch size, model vocabulary, etc. \n\nCould you consider initializing the ASR decoder with IndoBART as an additional comparison to account for the additional training data the cascade model may have seen? "}}, {"review_id": "mLc7mEEW3x", "reviewer": "Reviewer_PvXP", "summary": "", "questions": "Question A: Will you be releasing the corpus you have prepared for this task?\n\nQuestion B: Is sentence level the appropriate choice for evaluating structural ambiguity? For example, if the sentence \"I saw someone on the mountain.\" is preceded with \"I was about to take the trail leading up the mountain.\" then the context removes the ambiguity.\n\nQuestion C: Can you describe, generally, the motivation for this research? If it is only to make the text less ambiguous, are there particular situations when you see that as more valuable? If you believe this may contribute to downstream tasks, which ones?\n\nQuestion D: In line 369 you provided a survey to 20 participants for the human assessment. Did you ensure these participants are different than the crowdsourced participants mentioned earlier? Were these participants also crowdsourced?\n\nQuestion E: Were the participants in your study compensated for their contribution in any way? Were they fully informed of how their contributions would be used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for the insightful and constructive review. Below we address the reviewer's concerns point by point. \n1. **No ethics section, but there are ethical issues that deserve discussion. The paper makes use of numerous participants, at least some of which are crowdsourced. No information is given on their informed consent to participate or whether fair compensation was provided for their work. There is no ethics section which might clarify these points.**   \n  Answer: Before the experiment, all speakers, validators, and subjective evaluators have provided written consent for participating in this study in accordance with the institutional ethics guidelines reviewed and approved by the Institutional Review Board of the Research Ethics Committee of our institution. We have also explained how their contributions would be used and provide compensation based on the hourly work policy at our institute.\n  \n2. **When the corpus was created, participants were told to speak in such a way to make the intent of the speech unambiguous. This may lead to over-emphasis compared with natural speech. There was no mention of any evaluation of the data to avoid this.**   \n   Answer: To avoid over-emphasis, we have asked the speakers to keep speaking naturally. We also monitored each speaker during the recording and requested corrections if their speech sounded unnatural due to over-emphasis. Furthermore, before the recording session, we had provided the speakers with the structurally ambiguous texts that they would read, for them to study in advance. \n   \n3. **The corpus was created with only ambiguous sentences, and the non-ambiguous content was taken from another source. There is a chance that different recording qualities between news (LSCVR) and crowdsourced data could artificially raise the ability of the model to distinguish between ambiguous (tag 1 or 2) and non-ambiguous (tag 0) sentences.**   \n   Answer: Even if non-ambiguous utterances are ignored, our system maintains a commendable disambiguation accuracy of 0.74 and 0.78 for cascade and direct systems respectively.\n  \n4. **The amount of data used to train the text disambiguation model was significantly lower than the data used for training the end-to-end system. Given that the difference between the two proposed systems is only a few percentage points, it brings into question the conclusion that the direct model is clearly the better of the two (but they still are both demonstrably superior to the baseline).**   \n   Answer: The TD model is a pretrained IndoBART model which we finetuned with data described in Table 3. In addition, the ASR model and SD model were trained with the same data so it cannot be said that the cascade training data is less than the direct training data. However, we continued the training and found that the BLEU and ROUGE scores for the cascade system are differ slightly than the direct system, while the direct system's disambiguation accuracy is better than the cascade system with a difference of 2.5%. Thus, the direct system is just slightly better than the cascade system.\n  \n5. **It would be hard to reproduce the fine tuning of the IndoBART model without a little more information. Was it fine-tuned for a certain number of steps, for example?**   \n   Answer: We will include the usage of hyperparameters if the paper is accepted.  \n  The IndoNLG paper provides the optimal hyperparameters for fine-tuning IndoBART. Nonetheless, we still attempted a simple hyperparameter tuning for the learning rate and maximum epochs within each tagging method (baseline and cascade+tag). We explored three combinations of learning rates and maximum epoches, while keeping the other hyperparameters consistent with the settings specified in the IndoNLG paper. The result of the hyperparameter tuning are presented in the table below.\n    |               | Best learning rate | Best max epoch |\n    |---------------|--------------------|----------------|\n    | Baseline      | $10^{-4}$    | 10             |\n    | Cascade + tag | $10^{-4}$    | 50             |\n  \n\n6. **Will you be releasing the corpus you have prepared for this task?**   \n   Answer: If the paper is accepted, we will release the corpus.\n  \n7. **Is sentence level the appropriate choice for evaluating structural ambiguity? For example, if the sentence \"I saw someone on the mountain.\" is preceded with \"I was about to take the trail leading up the mountain.\" then the context removes the ambiguity.**   \n   Answer: While that idea might be valid, our study specifically concentrates on addressing structural ambiguity within a single sentence.\n  \n8. **Can you describe, generally, the motivation for this research? If it is only to make the text less ambiguous, are there particular situations when you see that as more valuable? If you believe this may contribute to downstream tasks, which ones?**   \n   Answer: As can be seen from the results of our human evaluation, native speakers can handle structurally ambiguous utterances quite well. However, non-native speakers may find this task challenging. In fact, the existing speech translation systems, both speech-to-text and speech-to-speech, often overlook ambiguity during translation. Therefore, this structural ambiguity resolution system has the potential to be used in speech translation systems, both speech-to-text and speech-to-speech.\n  \n9. **In line 369 you provided a survey to 20 participants for the human assessment. Did you ensure these participants are different than the crowdsourced participants mentioned earlier? Were these participants also crowdsourced?**   \n   Answer: Yes, all of the speakers and survey participants were crowdsourced, and there is no overlap between survey participants and speakers.\n  \n10. **Were the participants in your study compensated for their contribution in any way? Were they fully informed of how their contributions would be used?**   \n   Answer: Yes. All speakers, validators, and subjective evaluators have been provided written consent for participating in this study in accordance with the institutional ethics guidelines reviewed and approved by the Institutional Review Board of the Research Ethics Committee of our institution. We have also explained how their contributions would be used and provide compensation based on the hourly work policy at our institute.\n\nWe also appreciate your input on missing references, typos, grammar, style, and presentation improvements. If this paper is accepted, we will make the necessary enhancements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written in that it is clear to the approaches taken. It introduces a novel approach  and task in an under-served language for preserving the intent of speech in written form. The findings seem significant, even with a few methodological concerns.", "reasons_to_reject": "No ethics section, but there are ethical issues that deserve discussion (see the ethics section).\n\nAlso a few, mostly minor points:\n\n- When the corpus was created, participants were told to speak in such a way to make the intent of the speech unambiguous. This may lead to over-emphasis compared with natural speech. There was no mention of any evaluation of the data to avoid this.\n- The corpus was created with only ambiguous sentences, and the non-ambiguous content was taken from another source. There is a chance that different recording qualities between news (LSCVR) and crowdsourced data could artificially raise the ability of the model to distinguish between ambiguous (tag 1 or 2) and non-ambiguous (tag 0) sentences.\n- The amount of data used to train the text disambiguation model was significantly lower than the data used for training the end-to-end system. Given that the difference between the two proposed systems is only a few percentage points, it brings into question the conclusion that the direct model is clearly the better of the two (but they still are both demonstrably superior to the baseline).\n- It would be hard to reproduce the fine tuning of the IndoBART model without a little more information. Was it fine-tuned for a certain number of steps, for example?", "questions_to_authors": "Question A: Will you be releasing the corpus you have prepared for this task?\n\nQuestion B: Is sentence level the appropriate choice for evaluating structural ambiguity? For example, if the sentence \"I saw someone on the mountain.\" is preceded with \"I was about to take the trail leading up the mountain.\" then the context removes the ambiguity.\n\nQuestion C: Can you describe, generally, the motivation for this research? If it is only to make the text less ambiguous, are there particular situations when you see that as more valuable? If you believe this may contribute to downstream tasks, which ones?\n\nQuestion D: In line 369 you provided a survey to 20 participants for the human assessment. Did you ensure these participants are different than the crowdsourced participants mentioned earlier? Were these participants also crowdsourced?\n\nQuestion E: Were the participants in your study compensated for their contribution in any way? Were they fully informed of how their contributions would be used?", "paper_topic_and_main_contributions": "The paper addresses the problem of creating unambiguous interpretations in Indonesian transcribed speech, when then speech itself contains structural ambiguity (sentences which, as spoken/written, may have more than one meaning). To do this, they provide their model with prosodic information using specific signal processing techniques: root means square energy (RMSE) and mel-spectrogram to represent pauses; and Fundamental Frequency (F0) to represent pitch. To produce unambiguous text, three approaches are compared. The baseline approach uses ASR and a specially trained text disambiguation model. The proposed approaches use a modified ASR that attempts to tag one of three interpretations to the transcribed audio, then performs text disambiguation. The last approach is end-to-end. The paper demonstrates significant improvement with the proposed approaches over the baseline. The work also includes the preparation of a corpus for this task.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Lines 62 - you mention \"most of the studies\" it would be helpful to cite some examples\n\nLine 115 - while you may know this from your own experience, there are a number of works that could be cited to demonstrate how Indonesian NLP lags behind. Consider whether a citation here might strengthen your statement.", "Typos_Grammar_Style_and_Presentation_Improvements": "line 054 - resulting should be results\nline 073-075 - there seems to be a missing word in this sentence starting with \"However\"\nline 230 - \"how to build the system\" seems vague and broad. I'd suggest what you are doing here is looking for specific systems that might help to preserve the prosodic information\nTable 6 - second to last column in the \"Mel\" section is missing data.", "Justification_for_Ethical_Concerns": "The paper makes use of numerous participants, at least some of which are crowdsourced. No information is given on their informed consent to participate or whether fair compensation was provided for their work. There is no ethics section which might clarify these points."}}, {"review_id": "0CrpXEd6bp", "reviewer": "Reviewer_iZHD", "summary": "", "questions": "- [A] Why have the authors chosen end-to-end speech-former ASR models when there are only 40 hours (although seemed to have converged well looking at the results). I would have tried HMM-based ASRs or transfer learning + SSLs. Why was this model chosen over other good alternatives?\n- [B] How would the system perform if the TD model had direct access to prosody information. In other words, in addition to text that has tags, if we input some prosody features extracted from speech to the TD model, would it do better?\n- [C] No discussion or mention of the hyper parameters used in any of the setups (cascaded, direct). Is it all default hyperparameters? If so, could the authors add them in the appendix for easy reference. If not, could the authors show the results for all the hyperparameters they experimented with in the appendix?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for the insightful and constructive review. Below we address the reviewer's concerns point by point. \n1. **Done on read speech, which tends to be not so natural. But I can imagine conversational/spontaneous speech to be difficult to collect, especially for the task at hand**   \n   Answer: In our work, we start with read speech to reduce variation and focus on ambiguity. Even so, we still maintain the naturalness of our corpus by informing the speakers to keep speaking naturally and monitoring each speaker during recording so that we can ask the speakers to correct their reading if their speech sounds unnatural.\n  \n2. **Why have the authors chosen end-to-end speech-former ASR models when there are only 40 hours (although seemed to have converged well looking at the results). I would have tried HMM-based ASRs or transfer learning + SSLs. Why was this model chosen over other good alternatives?**   \n   Answer: Transformer has shown success in various natural language processing tasks, both in text and speech modalities. It can model long-range dependencies and be trained more effectively than recurrence sequence-to-sequence models. Therefore, we chose to use Transformer-based ASR to demonstrate that even with the limited data at our disposal, this powerful architecture can be used to train ASR effectively while still give adequate result.\n  \n3. **How would the system perform if the TD model had direct access to prosody information. In other words, in addition to text that has tags, if we input some prosody features extracted from speech to the TD model, would it do better?**   \n   Answer: Incorporating prosodic features extraction into the TD model appears to be a good idea. However, in this work, we have not done that yet. We plan to do that in future research. \n  \n4. **No discussion or mention of the hyper parameters used in any of the setups (cascaded, direct). Is it all default hyperparameters? If so, could the authors add them in the appendix for easy reference. If not, could the authors show the results for all the hyperparameters they experimented with in the appendix?**   \n   Answer: We will include the usage of hyperparameters if the paper is accepted.  \n   * The ASR and SD models training in this study did not involve hyperparameter tuning since it already employed almost identical hyperparameters to the Speech-transformer paper.\n   * The IndoNLG paper provides the optimal hyperparameters for fine-tuning IndoBART. Nonetheless, we still attempted a simple hyperparameter tuning for the learning rate and maximum epochs within each tagging method (baseline and cascade+tag). We explored three combinations of learning rates and maximum epoches, while keeping the other hyperparameters consistent with the settings specified in the IndoNLG paper. The result of the hyperparameter tuning are presented in the table below.\n     \n        |               | Best learning rate | Best max epoch |\n        |---------------|--------------------|----------------|\n        | Baseline      | $10^{-4}$    | 10             |\n        | Cascade + tag | $10^{-4}$    | 50             |\n\nWe also appreciate your input on typos, grammar, style, and presentation improvements. If this paper is accepted, we will make the necessary enhancements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is a delightful read (almost like an answer to Chrupała (2023)'s Putting \"Natural\" in Natural Language Processing, where he talks about going beyond just using speech to transcribe it to words)! A much needed new perspective on how speech can be used to solve difficult (using text-only methods) NLP tasks such as syntactic disambiguation.\n- Amazing to also see this done in Indonesian! Usually such new problems statements are first explored in English/high resource languages and then applied to other low-resource languages.\n- Nice comparison between cascaded systems and end-to-end systems\n- A novel dataset that furthers research in this direction", "reasons_to_reject": "- Done on read speech, which tends to be not so natural. But I can imagine conversational/spontaneous speech to be difficult to collect, especially for the task at hand.\n- Problem of disambiguation seems to be oversimplified, which models having only 2 or 3 tags (<0>, <1>, <2>).", "questions_to_authors": "- [A] Why have the authors chosen end-to-end speech-former ASR models when there are only 40 hours (although seemed to have converged well looking at the results). I would have tried HMM-based ASRs or transfer learning + SSLs. Why was this model chosen over other good alternatives?\n- [B] How would the system perform if the TD model had direct access to prosody information. In other words, in addition to text that has tags, if we input some prosody features extracted from speech to the TD model, would it do better?\n- [C] No discussion or mention of the hyper parameters used in any of the setups (cascaded, direct). Is it all default hyperparameters? If so, could the authors add them in the appendix for easy reference. If not, could the authors show the results for all the hyperparameters they experimented with in the appendix?", "paper_topic_and_main_contributions": "Speech to text systems usually transcribe the exact utterance spoken into words. However, speech is often a lot more than what words can capture and this is what the authors exploit. They test whether adding information from speech helps in syntactic disambiguation. For this, they create a novel dataset in a low-resource Indonesian language -- speech -> tagged sentences -> disambiguated sentences. \nThey do this in two ways: (1) cascaded system, where the ASR, along with words, outputs some tags. Next, a text disambiguation block is used to finally disambiguate the sentence using these tags (2) end-to-end system that takes speech as the input and outputs disambiguated text. The authors report the performance in terms of 3 metrics: BLEU, ROUGE, and disambiguated text accuracy. The cascaded system performs better in terms of ROUGE and BLEU scores while the direct system performs better in terms of disambiguated text accuracy.", "excitement": 5, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 488/489 “with” and “without”\n- Line 347/348 “which should have inherently preserves all..” doesn’t read well to me. \n- BLEU and ROUGE is reported in terms of % in the abstract, which is not the norm. "}}]

Author rebuttal:

Meta-review: While reviewers pointed out the size, naturalness of the dataset, simplification of the task, and minor ablation study, it seems the work verified itself with the importance of the task, well-designed experiment, original approach, and significant findings. The paper is well written and the evaluation metric is also reasonable to convince readers.

However, it still remains an ethical concern and would require additional evidence as suggested by ethics chairs.
Decision: Accept-Main