Semantic Role Labeling from Chinese Speech via End-to-End Learning (2024)

Abstract: Semantic Role Labeling (SRL), crucial for understanding semantic relationships in sentences, has traditionally focused on text-based input. However, the increasing use of voice assistants and the need for hands-free interaction have highlighted the importance of SRL from speech. SRL from speech can be accomplished via a two-step pipeline directly: transcribing speech to text via Automatic Speech Recognition (ASR) and then applying text-based SRL, which could lead to error propagation and loss of useful acoustic features. Addressing these challenges, we present the first end-to-end approach for SRL from speech, integrating ASR and SRL in a joint-learning framework, focusing on the Chinese language. By employing a Stright-Through Gumbel-Softmax module for connecting ASR and SRL models, it enables gradient back-propagation and joint optimization, enhancing robustness and effectiveness. Experiments on the Chinese Proposition Bank 1.0 (CPB1.0) and a newly annotated dataset AS-SRL based on AISHELL-1 demonstrate the superiority of the end-to-end model over traditional pipelines, with significantly improved performance.

heading: 1 Introduction
text: Semantic role labeling (SRL), as one of the core tasks in natural language processing (NLP, Gildea and Jurafsky, 2000), concentrates on analyzing semantic relationships between predicates and their corresponding arguments in sentences, such as who did what to whom, as well as when, where and how it occurred (Màrquez et al., 2008). Semantic information has been exploited in a variety of complex downstream NLP tasks including information extraction (Surdeanu et al., 2003; Niklaus et al., 2018), question answering (He et al., 2015;
* Authors contributed equally. † Corresponding author: Meishan Zhang.
Zhang et al., 2020), and robot command parsing (Thomason et al., 2020; Vanzo et al., 2020).
Previous SRL typically assumes input as text (Pradhan et al., 2005; FitzGerald et al., 2015; Fei et al., 2020; Campagnano et al., 2022). However, in many real-world scenarios, such as for the elderly who may find typing challenging, or in situations requiring remote control of machines, speech often emerges as the more direct medium for conveying information (Fowler, 1986; Bingol and Aydogmus, 2020). As voice assistants (Ahmed et al., 2022), social bots (Mahdi et al., 2022), and intelligent home devices (Torad et al., 2022) become increasingly prevalent, the demand for language understanding capabilities that can process speech inputs grows correspondingly. Thus, SRL from speech directly would exhibit increasing interest.
SRL from speech can be accomplished via a twostep pipeline directly: (1) automatic speech recognition (ASR) that transcribes the speech into text
8898
first, and then (2) text-based SRL. However, such a pipeline could be insufficient for speech-based SRL (Ghannay et al., 2018), as illustrated in Figure 1. On the one hand, the sequential pipeline might suffer from the commonly-recognized error propagation problem, i.e., the ASR errors would propagate to SRL (Hatmi et al., 2013; Simonnet et al., 2018), since traditional text-based SRL always assume well-written text as input. On the other hand, after ASR the acoustic features are completely isolated to the second step (Jannet et al., 2017), while part of them might be useful for SRL, e.g., the phonological features such as stress, pauses and undulations. To address the above issues, end-to-end learning of the two tasks is one natural solution.
In this work, we present the first work to tackle SRL from speech in an end-to-end way, uniting ASR and SRL in a joint-learning paradigm. We focus on the Chinese language as the pioneer study here. We start from the state-of-the-art ASR and SRL models, and then connect the two models by a Stright-Through Gumbel-Softmax (GS) module (Jang et al., 2017). The GS enables gradient backpropagation over the intermediate discrete transcribed texts, therefore a joint optimization towards the ASR and SRL objectives can be achieved, and the end-to-end learning is formed naturally. Additionally, as the transcribed text might be deformed into a noisy one, which makes gold-standard annotations difficult to be explored during SRL learning, we design an alignment mechanism to reach a reasonable SRL objective. This alignment also enables more robust evaluation for speech-based SRL and akin.
We conduct experiments on Chinese Proposition Bank 1.0 (CPB1.0) (Xue and Palmer, 2003) mainly to verify the effectiveness of our end-to-end model. Specifically, we recruit Mandarin-qualified annotators to obtain authentic speech for the test set for evaluation, and the speech for the remaining part is annotated by a text-to-speech (TTS) tool,1 since the manual annotation is of high cost. In addition, we also build another evaluation dataset AS-SRL based on the open-source Mandarin speech corpus AISHELL-1 (Bu et al., 2017), annotating it with SRL by experts following the guideline of CPB1.0. Results show that our end-to-end method is highly effective, achieving averaged improvements of 0.47% and 1.20% in the F1 score com-
1https://learn.microsoft.com/en-us/azure/aiservices/speech-service/text-to-speech
pared with the standard pipeline counterpart. We also design a series of other strong baselines for comparisons, and the results show that our final model outperform all these systems. Further, we perform in-depth analysis to provide a comprehensive understanding of our methodology.
In summary, our contributions can be summarized as follows:
• We present the first study of SRL from Chinese speech and construct the first real-world Chinese speech-based SRL dataset. • We propose an end-to-end approach for speech-based SRL and utilize alignment tools to achieve robustness to transcription noise. • We conduct extensive experiments to compare our method with several pipeline methods, and results show that our method can advance the speech-based SRL further.
Our code and datasets will be publicly available at github/DreamH1gh/SpeechSRL to facilitate future research.

heading: 2 Method
text: A straightforward solution for SRL from speech is through a pipeline strategy, first executing ASR and then performing SRL. Since the pipeline architecture induces error propagation inevitably and meanwhile cannot use acoustic clues for SRL, we suggest end-to-end learning to unite the two tasks, as shown in Figure 2. This section first introduces a state-of-the-art pipeline, and then describes our end-to-end learning approach.

heading: 2.1 Pipeline
text: We use the transformer-based sequence-tosequence Whisper (Radford et al., 2023) model for ASR and a word-based graph parsing model introduced by Zhou et al. (2022) for SRL. Both models are representative systems with top performance for ASR and SRL, respectively.
Automatic Speech Recognition. Given a piece of raw speech, ASR is to convert it into the corresponding transcription X = (c1, . . . , cn), where c denotes the character. Specifically, the Whisper model first uses a classical feature extractor to process the raw input speech, obtaining a sequence of speech features S = (s1, . . . , sN ) (commonly, N ≫ n). Then, S is fed into a transformer encoder, reaching high-level hidden states Hasr = (hasr1 , . . . ,h asr N ). Subsequently, a
transformer-based decoder is used to score the output tokens sequentially based on previous predications and the hidden states Hasr, finally resulting in Oasr = (oasr1 , . . . ,o asr n ). The whole process can be formalized as follows:{ Hasr = Transformer-Encoder(S), Oasr = Transformer-Decoder(Hasr). (1)
During training, the cross-entropy objective is used for the ASR model parameter optimization: { P asr : pasr1 , . . . ,p asr n = softmax(O asr),
Lasr = − ∑n i=1 ci log p asr i [ci],
(2)
where pasri [ci] indicates the probability of ith token being ci and ci is the one-hot vector form of ci. During inference, we output the max-scored characters based on Oasr.
Note that the Whisper model follows a standard transformer-based sequence-to-sequence neural architecture. Its strong capability of ASR is mainly sourced from the large-scale training corpus collected by the original work. Here we directly exploit the well-trained parameters of Whisper for ASR. For more training details, one can refer to their paper.
Semantic Role Labeling. Zhou et al. (2022) treat SRL as graph parsing, first converting predictargument structures into a dependency graph, and then exploiting a well-designed parser to achieve the goal. The graph is built simply by connecting the boundaries of semantic roles with their predicates, and predicates are associated with a special root symbol. In our pipeline, the transcription X is
a character sequence in Chinese, thus we need to convert the original word-based dependency graph into characters. The conversion method is simple and direct, with details provided in A.1.
The overall character-based SRL parsing model can be also illustrated as an encoder-decoder architecture. First, the sentential characters X = (c1, . . . , cn) go through a pretrained language model (PLM), obtaining the well-represented encoder output Hsrl = (hsrl1 , . . . ,h srl n ). Then a graph decoder is executed. For each candidate dependency edge <i, j> (i is the head and j is the modifier), a biaffine operation is adopted for scoring. After the edges are ready, a set of label-aware biaffine operations can be applied to calculate the label scores. The full process can be formalized as follows:   Encoding:Hsrl = P̃LM(X), Edge <i, j> : hhi ,h m j = MLP h(hsrli ),MLP m(hsrlj ), o edge i,j = [h h i ] TWEh m j + [bE] Thmj Label <l|i, j> : h̄hi , h̄ m j = MLP h (hsrli ),MLP m (hsrlj ),
olabell|i,j = [h̄ h i ] TWL,lh̄ m j + [bL,l] T h̄mj .
(3)
where [·]T indicates matrix transposition.
To train the above model, we also use crossentropy loss as objective. Essentially, the edge prediction is a binary classification of whether an edge being retained in graph, while the label prediction belongs to multi-label classification. As a
result, our SRL objective is defined as:    p edge i,j = sigmoid(o edge i,j ), plabell|i,j = softmax(o label ∗|i,j)[l], Lsrl = − ∑ <l|i,j>∈G log(p edge i,j × plabell|i,j )
−∑ <i,j>/∈G log(1− pedgei,j ),
(4)
where G is the gold-standard graph of SRL. During inference, the edges with probability larger than 0.5 are kept, and the max-scored label is assigned to a given dependency edge. The decoding graph might be incompatible with SRL, which is addressed by constrained decoding. For the details, one can refer to the original paper.2

heading: 2.2 End-to-End Learning
text: The above pipeline isolates ASR and SRL completely, making the two components incapable of interaction. Here we introduce a simple end-toend learning framework to connect the two tasks. The key idea is to propagate the objective loss of SRL directly into ASR, not only letting SRL robust to the ASR errors but also enabling SRL to use acoustic features directly. The major difficult lies in that the discrete ASR outputs disable the back-propagation of SRL training gradients into the ASR component. To address this problem, StrightThrough Gumbel-Softmax is exploited here (Jang et al., 2017).
Gumbel-Softmax. As mentioned in Sec 2.1, we use the widely-accepted argmax operation for nextcharacter prediction to obtain the transcribed characters, which are then delivered to SRL as input for training. This operation leads to non-differentiable variables, which is the source of the isolation between ASR and SRL. Different from argmax where the resulted one-hot vector is deterministic, GS introduces randomness into the prediction, resulting in a sampled vector. More importantly, GS exploits a reparameterization trick to make the sampled vector differentiable. Since the added noise follows a Gumbel distribution, the resulting next-token prediction distribution is essentially consistent with the original distribution.
Concretely, given a speech input S, we feed it into the same Whisper encoder-decoder module, obtaining the output scores Oasr of the transformerdecoder as shown in Equation 1. During training,
2Here we only take the first-order scoring to introduce the SRL model for brevity. The higher-order scoring can be achieved by TriAffine as mentioned by their original paper.

heading: Trans:
text: we exploit the GS operation to calculate next-token distributions instead of softmax:
p gs i,j =
exp ( (oasri,j + gi,j)/τ ) ∑Casr
k=1 exp ( (oasri,k + gi,k)/τ ) , (5)
where gi,∗ are i.i.d samples drawn from Gumbel(0, 1)3 , τ is the temperature factor and Casr denotes the output vocabulary size of the ASR model. Then, gumbel sampling is used instead of the argmax operation to obtain the one-hot output at each nexttoken prediction, and finally we aggregate the ASR outputs, resulting in Xasr = (c1, . . . , cn).
Note that the sampled one-hot outputs Xasr = (c1, . . . , cn) should not be converted into characters by the ASR output vocabulary, otherwise the differentiability would be lost. Therefore we need to feed them into the SRL encoder directly. However, there is a mismatch between the character vocabularies of ASR output and SRL input. To solve this issue, we maintain a matrix M to implement the transformation, which is defined as follows:
Mi,j = { 1, if Vasr[i] = Vsrl[j], 0, otherwise,
(6)
where Vasr and Vsrl denote the character vocabularies of ASR output and SRL input, respectively. Using M , we can get the input one-hot vectors of SRL by Xsrl = XasrM . After that, we feed Xsrl into the encoding part P̃LM as shown in Equation 3, and then ASR and SRL connect naturally.
SRL Alignment. Through the GS module, we connect ASR and SRL together for training. However, there still exists one big problem. The sam-
3The Gumbel(0, 1) distribution can be sampled using inverse transform sampling by drawing µ ∼ Uniform(0, 1) and computing g = −log(−log(µ)).
pled vector of ASR can often differ from the wellwritten gold-standard text for SRL training, as shown in Figure 3. When the phenomenon happens, how to construct the oracle character-based dependency graph in terms of the original goldstandard graph?
To solve the problem, we suggest a characterlevel alignment tool to project the original goldstandard graph into the sampled text. The idea is borrowed from the line of annotation projection in cross-lingual dependency parsing (Schuster et al., 2019; Zhang et al., 2019; Ozaki et al., 2021; Zhang et al., 2023). We can use a similar projection strategy since SRL is modeled as dependency parsing here. High-quality projections would be expected as the noisy text by GS is mostly similar to the original one.
Concretely, there are two steps. First, we obtain a high-performance character alignment tool. by using the unsupervised fastAlign4. A large-scale parallel corpus is constructed automatically with no laborious cost by mocking the GS sampling, which is used to train the fastAlign. Second, we project the character-level SRL dependencies from the source text into the target noisy text one by one. The process is very easy since accurate alignments can be obtained in most cases because very few ambiguities exist in our scenario. In cases of exceptions the dependencies are discarded directly.
Training and Inference. For training, we adopt the SRL objective as the training loss named Ljoint, which can jointly update the ASR module with the SRL module at the same time. For inference, there is no difference to the pipeline.

heading: 3 Experiments

heading: 3.1 Settings
text: Dataset. Our experiments are evaluated based on two Chinese datasets: CPB1.0 (Xue and Palmer, 2003) and AISHELL-1 (Bu et al., 2017). CPB1.0 is a widely used Chinese SRL dataset and contains gold-standard text and corresponding SRL labels. For the test set, we recruit Mandarin-qualified annotators from colleague to obtain authentic speech for evaluation. The training and development sets are annotated by Microsoft Azure TTS tool because of the high cost of manual annotation. AISHELL-1 is a widely used open-source Mandarin speech corpus, containing authentic speech and gold-standard
4https://github.com/clab/fast_align
text. Due to the extensive size of AISHELL-1, we randomly select 9,000 speech-text pairs and invite experts with a proficient understanding of the SRL task to annotate them with SRL following the guideline of CPB1.0. The statistics about the two datasets are illustrated in Table 1. The specific annotation process details are placed in A.2 for limited space.
Evaluation Metrics. Consistent with prior research of Chinese ASR, we employ the character error rate (CER) to evaluate the performance of ASR. For SRL, we measure the performance by the atomic predicate-argument structure, i.e., a tuple of <predicate, argument, relation>. We regard a tuple as correct only when the predicate, argument and relation are all exactly matched with the goldstandard answers. The precision, recall as well as their F1 score are calculated.
Noticeably, the first-step ASR errors can make the SRL metric calculation problematical, because we cannot determine the matching status by the difference between the ASR outputs and the desired gold-standard text, as shown in Figure 3. To solve the issue, we perform a character-level alignment among the two texts before the metric computation. We achieve this goal by our well-trained fastAlign tool followed with a human checking. In fact, we find that the human checking can be almost omitted as the fastAlign can have an accuracy higher than 99% during this particular alignment.
Model Details. Regarding Whisper, we select large-v25 (Radford et al., 2023), which performs well on ASR. Particularly, we performs a continual pretraining on the training set to adapt Whisper for our benchmark datasets. By default, the Whisper parameter updating is achieved using LORA (Hu et al., 2022) for efficient learning. We utilize bert-
5https://huggingface.co/openai/whisper-large-v2
base-chinese6 (Devlin et al., 2019) to obtain text embeddings for our SRL model. For convenience, we denote the baseline SRL model as SRLGParser all through this work.
Hyperparameters. All experiments are conducted using a single NVIDIA A100 Tensor Core GPU (40G) card. We run each setting by 5 times with different random seeds, and the median evaluation scores are reported. For the setting of the SRL model parameters, we follow the settings of (Zhou et al., 2022). The temperature factor in Equation 5 is set as τ = e−5. We finetune the model for 20 epochs using learning rate of 3e-5, and select the best iteration number according to the development results.

heading: 3.2 Main Results
text: Table 2 shows the main results on CPB1.0 and AS-SRL. It can be observed that our proposed endto-end method achieves better results than the standard pipeline across almost all settings. Our final model, i.e., Whisper ◦ SRLGParser that optimizes with both ASR and SRL objectives in an end-toend way, can lead to improvements of 0.47% and 1.20% on the two datasets,7 respectively. Following, we make comprehensive comparisons between the pipeline and end-to-end models.
First, we examine the SRL performance of SRLGParser in terms of auto ASR transcriptions and gold texts as inputs under the pipeline architecture. We can see that the CER values of Whisper are 3.16 and 4.45 for CPB1.0 and AS-SRL, respectively, which is highly competitive. Compared with
6https://huggingface.co/bert-base-chinese 7Both improvements are significant with a p-value below
10−5 under pairwise t-test.
gold-text inputs, the ASR errors can bring significant decreases of 81.96−76.64 = 5.32 for CPB1.0 and 79.87− 72.80 = 7.07 for AS-SRL. This suggests that the quality of the ASR text impacts the SRL performance much.
It is worth noting that the end-to-end learning procedure can be regarded as a special case of data augmentation. As illustrated in Sec 2.2, the gumbel sampled texts together with their oracle SRL annotations provide a new source of training instances. To test the effectiveness as data augmentation, we simply dump all augmented instances and add them into the pipeline SRL training, as denoted by Whisper + SRLGParser (+GS AUG). As shown, we can see that significant improvements 73.94−72.80 = 1.14 can be achieved on AS-SRL, whereas the increase on CPB1.0 is very marginal, where the possible reason could be that the TTSgenerated speech leads to little variation during GS sampling. The observation indicates that GS could be one prospective strategy for data augmentation.
Whisper ◦ SRLGParser (gold SRL) denotes that only when the ASR transcription exactly matches the gold-standard text, the SRL objective is calculated. This is an easy solution of end-to-end learning without SRL alignment. Even in this case, the joint learning can bring better performance, which further demonstrates the effectiveness of our end-to-end learning approach. Since no extra SRL instance is exploited in comparison with the pipeline, the performance gains indicate that acoustic features are useful for SRL from speech. By comparing Whisper ◦ SRLGParser (gold SRL) with the final model, we can again recognize the effectiveness of the augmented SRL instances by GS sampling. On both CPB1.0 and AS-SRL datasets,
we can observe appreciable improvements.

heading: 3.3 Analysis
text: The quality of alignment tool. The quality of the alignment tool plays a vital role in end-to-end SRL from speech. Therefore we conduct human evaluation to check the alignment accuracy. Specifically, during our SRL-performance evaluation, we have collected such data: first automatic alignment by our tool and then human collection. We take the collected data from the evaluation of our final model as an example (i.e., Whisper ◦ SRLGParser). Table 3 reports the results. We can see that the manual correction can be almost omitted, since the proportion of corrected characters is only 0.83% for CPB1.0 and 0.42% for AS-SRL, yet most of them might be irrelevant with SRL evaluation. The high alignment accuracy also supports the high-quality of our SRL annotation projection greatly.
The sensitivity of GS temperature to our endto-end learning. The GS temperature plays an important role in sampling. When the value is close to 1, GS would be almost equivalent to the standard softmax. We conduct experiments on the values of temperature from e−3 to e−6 to observe whether the model is sensitive to the temperature setting. The final model Whisper ◦ SRLGParser is employed for verification. Figure 4 shows the
results. We can see that the model can be influenced by the factor. As the value approaches to 1, the performance decreases significantly. However, the the value is lower than 10−5, the performance also a downtrend. As τ nears 0, the output distribution would become a meaningless one-hot distribution.
Fined-grained performance by the concrete SRL roles. Previous evaluation mainly focuses the overall tuple-level metrics. Here we look into the fined-grained SRL results in terms of different SRL roles, including predicates, core SRL arguments such as ARG0-2, and other SRL arguments such as TMP, LOC and etc. We compare our final endto-end learning model with the pipeline system. Figure 5 shows the results on our investigated two datasets. As shown, the predicates and the core semantic arguments always have a better performance than others, since these roles could be fully trained because of their high occurrence. As expected, the end-to-end learning can give better performance for almost all type of SRL roles. Moreover, the scores for predicate recognition and role labeling recognition on the AS-SRL dataset improves more significantly, suggesting that the the introduction of authentic speech features is more capable of facilitating the model to recognize the arguments and its roles better than TTS speech features. On the one hand, since the predicates are stressed most of the time, the accuracy of recognizing predicates improves on the authentic speech AS-SRL dataset by 0.44%. On the other hand, the role Others in both datasets are substantially improved, as Chinese has a variation of tones and
our model becomes more sensitive to recognizing location, time, etc.

heading: The ASR alterations after our end-to-end
text: learning. According to our experimental results, we ca see that our best model has a slightly higher CER than the pipeline model Whisper + SRLGParser, but the SRL results are all better than the method of pipeline. We analyze the differences between our model and the results transcribed by the pipeline model, and find that our model have fewer predicate transcription errors. For example, in the sentence “涉及人才管 理(involve talent management)” , the predicate “涉 及(involve: she ji)” is not transcribed correctly by Whisper, which transcribes it as “设计(design: she ji)” , whereas the Whisper after our end-toend learning, is transcribed successfully. While we find sometimes the Whisper can produce SRLirrelevant errors, e.g., the adjacent words such as “高新(high-tech: gao xin)” which could be correctly recognized by our baseline but confuse to “高薪(high-pay: gao xin)” by our end-to-end model, under a context “高新开发区(high-tech development zone)”.

heading: 4 Related Work
text: Semantic role labeling was originally conceptualized as a task for shallow natural language understanding by Gildea and Jurafsky (2000). The conventional approach to SRL involves two subtasks: (1) predicate identification followed by (2) argument role labeling (Swanson and Gordon, 2006; Punyakanok et al., 2008; Roth and Lapata, 2016; Marcheggiani and Titov, 2017; Strubell et al., 2018; Xia et al., 2019). In some studies, SRL task is decomposed with the goal of improving argument role labeling performance when predicates are provided as gold (He et al., 2018; Tan et al., 2018; Ouchi et al., 2018; Li et al., 2018; Fei et al., 2021). Recently, there has been an increasing interest in consolidating the two subtasks by a single model (Zhou et al., 2022). The line of work completes SRL fully with SOTA performance, and meanwhile exhibits significant efficiency advantages. Therefore, we follow this line of work, selecting (Zhang et al., 2022; Zhou et al., 2022) as the SRL baseline.
ASR has been concerned since very early (Lee et al., 1990; Huang et al., 1993; Chou et al., 1994). Deep learning has advanced this task greatly, with models like LAS employing RNNs to capture the contextual nuances of speech signals, leading to
notable enhancements (Chan et al., 2015; Amodei et al., 2016; Sriram et al., 2018). Moreover, the breakthrough success of the Transformer architecture in various NLP tasks has inspired the integration of attention mechanisms into ASR, further advancing its capabilities (Tang, 2023; Dong et al., 2023a). Whisper employs a Transformer-based architecture, trained on large-scale corpora (Radford et al., 2023). It has essentially achieved SOTA performance on ASR tasks. Thus, Whisper is selected as the model for our exploration of ASR.
The adventure ASR technologies (Cheng et al., 2023; Radford et al., 2023) and the increasing availability of large-scale speech datasets (Bu et al., 2017; Zhu et al., 2019; Wang et al., 2021) have prompted a transition towards integrating speech directly into NLP tasks. Many studies, addressing fundamental NLP tasks such as sentiment analysis (Shon et al., 2021; Aicher et al., 2022), machine translation (Ren et al., 2020; Fang and Feng, 2023b,a), and spoken language understanding (Seo et al., 2022; Dong et al., 2023b), have adeptly leveraged speech, employing end-to-end approaches to achieve remarkable performance by exploiting intrinsic features of speech. However, for more intricate NLP tasks like parsing (Caines et al., 2017; Jamshid Lou et al., 2019; Pupier et al., 2022; Lai et al., 2023; Tseng et al., 2023; Shao et al., 2023), pipeline models that convert speech to text before applying conventional NLP strategies are commonly used. Our study pioneers SRL from speech, proposing an end-to-end framework, marking a novel approach in this field.

heading: 5 Conclusion
text: In this paper, we proposed a novel approach to SRL from speech through an end-to-end methodology, integrating ASR and SRL into a unified learning framework. This work, pioneering in its focus on the Chinese language, leverages state-of-the-art ASR and SRL models, interconnected via a StrightThrough Gumbel-Softmax module. The GS module facilitates gradient back-propagation across intermediate, discrete transcriptions, enabling joint optimization of ASR and SRL objectives and the natural formation of end-to-end learning. To address the challenge of transcriptions potentially becoming noisy, an alignment tool is devised, ensuring that SRL learning can effectively utilize goldstandard annotations and allowing for a more robust evaluation of speech-based SRL.
We constructed two Chinese speech-based SRL datasets and performed experiments on them to evaluate our end-to-end method. Our method not only demonstrates significant performance improvements over traditional pipeline methods but also outperforms a series of strong baseline models. Further in-depth analysis provides a comprehensive understanding of the approach, illustrating its potential for broader application and setting a new benchmark in the field.

heading: Limitations
text: Our work has four major limitations. First, we only conducted our experiments on Chinese datasets. While our approach is applicable to other languages, there is a lack of experimental validation in these cases due to the time-consuming and high cost of data annotation. In the future, we will validate the effectiveness of our method on other languages. The second is that our experiments were confined to traditional classification-based SOTA SRL methods. Large language models (LLMs) excel in numerous NLP tasks. However, they currently fall short in SRL tasks and present optimization challenges due to their high computational costs. Nonetheless, investigating the integration of LLMs into speech-based SRL remains a promising research direction. We plan to thoroughly investigate the effectiveness of end-to-end SRL utilizing LLMs in future. Moreover, the training data of the ASR model we used (Whisper) we used is not publicly known, so it is uncertain there are no data leakage issues affecting ASR performance in the experiments. We will experiment on open source ASR models in the future. The last one is that ASR tasks in experiments (laboratory speech) may be easier than in the real world. Real-world speech contains more noise, and we plan to validate the effectiveness of our method on real-world speech.

heading: Ethical Statement
text: We construct two datasets for Chinese speechbased SRL. All instances in the experiment are from two existing datasets: CPB1.0 and AISHELL1, which are publicly available and do not contain sensitive and harmful information. We have obtained the license for these datasets. During recruiting annotators, we claim that all potential participants have the freedom to choose whether to partake in the study, and they may withdraw at any time without facing any negative consequences. All
annotators are properly compensated under open market competition. Moreover, the entire annotation task is conducted anonymously, with no connection to any private information of the annotators. We have not made any changes to the annotation results. Overall, the establishment of our datasets is compliant with ethics.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful comments. This work is supported by the National Natural Science Foundation of China(NSFC) Grant Nos.U23B2055 and Nos.62176180.

heading: A Appendix

heading: A.1 Character-based Dependency Graph
text: SRLGParser is a word-based graph, which also needs to take into account its positional information
when connecting arguments to obtain a complete argument span. It proposes four attach schemata to build the graph, and we choose the BES schema by its default. The BES schema uses "B/E/S-r" as the label, indicating the beginning, end, and single. Given a sentence that made up of words, first add a "Root" node to the front of the sentence, and it points to all predicates. The relation between a predicate and an argument is represented by the edge. If the argument is just a word, the predicate is directly points to the argument, and its label is "Sr". Otherwise, the predicate points to the head and tail of the argument span, and its labels are "B-r" and "E-r" respectively. This can form a word-based dependency graph as in Figure 6 (a).
It is worth noting that the predicate for wordbased SRL only consists of a single word, however, for character-based SRL, the predicate may consist of multiple characters. In order to solve the problem that the original SRLGParser cannot recognize the predicate span, we add a new label "E-PRD" to indicate the tail of predicate. Unlike constructing word-based graph, in character-based graph, when the predicate is composed of multiple characters, "Root" points to the head of the predicate span, and the relation between the predicate and the argument is reflected by the connection between the head of
the predicate and the argument, and additionally, the connection to the tail of the predicate. The result is shown in Figure 6 (b).

heading: A.2 Annotation Process
text: In this work, we annotate two speech-based Chinese SRL datasets. To ensure that we obtain highquality labeled datasets, we undertake the following procedures.
For CPB1.0, we recruit three college students with Mandarin proficiency (level 2-A or above) to serve as annotators. The test set is evenly divided among them for reading and speech recording. While each annotator reads the text, the other two annotators act as listener to check whether the current speech can be understood correctly. Otherwise, the speaker will be asked to read the text again until it could be understood correctly.
For AS-SRL, the entire annotation workflow is divided into three stages: (i) developing a userfriendly online tool for annotators; (ii) establishing standard annotation guidelines based on the CPB1.0 rules; (iii) recruiting some experts to annotate the dataset, ensuring that each instance receives three annotations.
First, we use the label-studio platform to design an annotation platform for the SRL task. Then, we organize the CPB1.0 label definitions and selected 100 pairs of CPB1.0 data as examples to develop annotation guidelines.
Next, three university students specializing in Chinese linguistics are recruited for annotation. Before commencing the task, annotators are required to thoroughly review the annotation process guide and attempt annotating some data. Official annotation begins once their annotation results meet the
required standards. We ask two of them to annotate the data. Upon completion of annotation, a proofreading process is conducted and the consistency rate between the two annotators is 86%. Then, the third annotator check the different labeling results to decide which one to use. If all three have different opinions, the results will be sent to experts for correction.
Finally, we obtain AS-SRL, with the semantic role label distribution as shown in in Table 4. Note that in the annotation process, we follow the annotation method of CPB1.0 span-based SRL, i.e., we first annotate the spans corresponding to predicates and arguments, then annotate the corresponding relations, and finally converting these annotations into the dependency graph format.

heading: Future Work
text: In the future, we will validate the effectiveness of our method on other languages. The second is that our experiments were confined to traditional classification-based SOTA SRL methods. Large language models (LLMs) excel in numerous NLP tasks. However, they currently fall short in SRL tasks and present optimization challenges due to their high computational costs. Nonetheless, investigating the integration of LLMs into speech-based SRL remains a promising research direction. We plan to thoroughly investigate the effectiveness of end-to-end SRL utilizing LLMs in future. Moreover, the training data of the ASR model we used (Whisper) we used is not publicly known, so it is uncertain there are no data leakage issues affecting ASR performance in the experiments. We will experiment on open source ASR models in the future. The last one is that ASR tasks in experiments (laboratory speech) may be easier than in the real world. Real-world speech contains more noise, and we plan to validate the effectiveness of our method on real-world speech