Growing Trees on Sounds: Assessing Strategies for End-to-End Dependency Parsing of Speech (2024)

Abstract: Direct dependency parsing of the speech signal –as opposed to parsing speech transcriptions– has recently been proposed as a task (Pupier et al., 2022), as a way of incorporating prosodic information in the parsing system and bypassing the limitations of a pipeline approach that would consist of using first an Automatic Speech Recognition (ASR) system and then a syntactic parser. In this article, we report on a set of experiments aiming at assessing the performance of two parsing paradigms (graphbased parsing and sequence labeling based parsing) on speech parsing. We perform this evaluation on a large treebank of spoken French, featuring realistic spontaneous conversations. Our findings show that (i) the graph-based approach obtain better results across the board (ii) parsing directly from speech outperforms a pipeline approach, despite having 30% fewer parameters.

heading: 1 Introduction
text: Dependency parsing is a central task in natural language processing (NLP). In the NLP community, it has mostly been addressed on textual data, either natively written texts or sometimes speech transcriptions. Yet, speech is the main form of communication between humans, as well as arguably one of the most realistic types of linguistic data, which motivates the design of NLP systems able to deal directly with speech, both for applicative purposes and to construct corpora annotated with linguistic information. When parsing speech transcriptions, most prior work has focused on disfluency detection and removal (Charniak and Johnson, 2001; Johnson and Charniak, 2004; Rasooli and Tetreault, 2013; Honnibal and Johnson, 2014; Jamshid Lou et al., 2019), in an effort to ‘normalize’ the transcriptions and make them suitable input for NLP systems trained on written language. Using only transcriptions as input is a natural choice from an
225
NLP perspective: it makes it possible to use offthe-shelf NLP parsers ‘as is’. However, predicted transcriptions can be very noisy, in particular for speech from spontaneous conversations. Furthermore, transcriptions are abstractions that contain much less information than the speech signal. The prosody, and the pauses in the speech utterances are very important clues for parsing (Price et al., 1991) that are completely absent from transcriptions. Hence, we address speech parsing using only the speech signal as input. With the popularization of self-supervised method and modern neural network architecture (pretrained transformers), both speech and text domains now use similar techniques (Chrupała, 2023). This convergence of methodology has raised interest in other applications of speech models to go beyond ‘simple’ speech recognition. Thus, addressing classical NLP tasks directly on speech is a natural step and design NLP tools able to deal with spontaneous speech, arguably the most realistic type of linguistic production. In short, Our contributions are the following:
• we introduce a graph-based end-to-end dependency parsing algorithm for speech; • we evaluate the parser on Orféo, a large treebank of spoken French that features spontaneous speech, and compare its performance to pipeline systems and to a parsing-as-tagging parser; • we release our code at https://github. com/Pupiera/Growing_tree_on_sound.1

heading: 2 Parsers and pre-trained models
text: We define speech parsing as the task of predicting a dependency tree from an audio signal corresponding to a spoken utterance.2
Our parser is composed of 2 modules (Figure 1a): (i) an acoustic module that is used to predict transcriptions and a segmentation of the signal in words and (ii) a parsing module that uses the segmentation to construct audio word embeddings and predict trees.
Word level representations from speech To extract representations from the raw speech, we use a pre-trained wav2vec2 model trained on seven thou-
1The code is also archived at https://doi.org/10. 5281/zenodo.11474162.
2For the sake of simplicity, we will use the term ‘sentence’ in the rest of the article, even though the very definition of a sentence is debatable in the spoken domain.
sand hours of French speech: LeBenchmark7K3 (Parcollet et al., 2024). Parsing requires word-level representations. We use the methodology of Pupier et al. (2022) to construct audio word embeddings from the implicit frame level segmentation provided by the CTC speech recognition algorithm (Graves et al., 2006). The method consists in combining the frame vectors corresponding to a single predicted word with an LSTM.
Graph-based parsing We use the audio word embeddings –whose construction is described above– as input to our implementation of a classical graph-based biaffine parser (Dozat and Manning, 2016): (i) compute a score every possible arc with a biaffine classifier and (ii) find the best scoring tree with a maximum spanning tree algorithm.
Sequence labeling The sequence labeling parser follows Pupier et al. (2022) and is based on the dep2label approach (Gómez-Rodríguez et al., 2020; Strzyz et al., 2020), specifically the relative POS-based encoding (Strzyz et al., 2019). This method reduces the parsing problem to a sequence labeling problem. The head of each token is encoded in a label of the form ±Integer@POS. The integer stands for the relative position of the head considering only words of the POS category. Eg., -3@NOUN means that the head of the current word is the third noun before it.

heading: 3 Dataset
text: We use the CEFC-Orféo treebank (Benzitoun et al., 2016), a dependency-annotated French corpus composed of multiple subcorpora (CLESTHIA, 2018; ICAR, 2017; ATILF, 2020; Mathieu et al., (2012- 2020; André, 2016; Carruthers, 2013; Cresti et al., 2004; DELIC et al., 2004; Francard et al., 2009; Kawaguchi et al., 2006; Husianycia, 2011), and released with the audio recordings. The treebank consists of various types of interactions, all of which feature spontaneous discussions, except for the French Oral Narrative corpus (audiobooks). Orféo features many types of speech situations (eg. commercial interactions, interviews, informal discussions between friends) and is the largest French spoken corpus annotated in dependency syntax. The annotation scheme has been designed specifically for Orféo (Benzitoun et al., 2016) and differs from the Universal Dependency framework in many re-
3https://huggingface.co/LeBenchmark/ wav2vec2-FR-7K-large
gards (in particular: its POS tagset is finer-grained, whereas the syntactic function tagset has only 14 relations). The syntactic annotations of Orféo were done manually for 5% of the corpus and automatically for the rest of the corpus. The train/dev/test split we use makes sure that the test section only contains gold annotations. Nevertheless, the subcorpora with gold syntactic annotations correspond to low-quality recordings, which makes them a very challenging benchmark.

heading: 4 Experiments
text: Experimental settings Our experiments aim at: (i) comparing our graph-based parser to the seq2label model, (ii) comparing to pipeline approaches with text-based parsers, and (iii) assessing the robustness of word representations with control experiments: using word boundaries (provided in the corpus) as input for the audio models and gold transcriptions for the text-based model. We compare the following settings (illustrated in Figure 1):
• AUDIO: Access to raw audio only, the model creates word-level representation from the acoustic model as described in Section 2. • ORACLE: Access to raw audio and silver4 word-level timestamps, making it easier to create word representations and mitigating the impact of the quality of the speech recognition on parsing. • PIPELINE: Access to predicted transcriptions from the acoustic model only, then a language model uses the transcriptions as input for parsing. The training trees are modified to take into account any deletion and insertion of words. However, as for the speech approach, deletion or insertion penalizes the global score of the model since the model is evaluated against the gold transcriptions and not the modified one. The drawback of this approach is that no information about prosody or pauses is available. • TEXT: Access to gold transcriptions: this unrealistic setting provides an upper bound performance in the ideal case (perfect ASR).
Both PIPELINE and TEXT settings use a French BERT model: camembert-base5 (Martin et al., 2020) to extract contextualized word embeddings.
4The corpus contained word-level timestamps that have been automatically constructed through forced alignment.
5https://huggingface.co/almanach/ camembert-base
For PIPELINE and TEXT settings, on top of our implementations, we use hops (Grobol and Crabbé, 2021), an external state-of-the-art graph-based parser. The hops parser uses a character-bi-LSTM in addition to BERT to produce word embeddings, whereas our implementation does not (in an effort to make both versions of our parser, text-based and audio-based, as similar as possible).
Each parsing method for each modality is trained with the same number of epochs, the same hyperparameters (see Table 4 and 5 of Appendix A), and approximately the same number of parameters. We select the best checkpoint on the development set in each setting for the final evaluation. Our implementations use speechbrain (Ravanelli et al., 2021).
Metrics We use classical evaluation measures: Word Error Rate (WER) and Character Error Rate (CER) for speech recognition, POS accuracy (POS), Unlabeled Attachment Score (UAS), and Labeled Attachment Score (LAS) for dependency parsing.
We report results in Table 1 for the full corpus, and in Table 2 for a sub-corpus of the test set (Valibel) for which speech recognition is easier.
Evaluation To evaluate our architecture, we use a modified version of the evaluation script provided by the CoNLL 2018 Shared Task.6 The main limitation of this evaluation protocol is that it requires the two sequences to be exactly the same, which is not the case when speech recognition is involved. Thus, we modify this evaluation script to work even when the two sequences to evaluate are not of the same length. However, the modified script requires an alignement between the 2 sequences. For our purpose, we use an alignment based on edit distance, i.e. the same alignment strategy already used to compute WER.
The modified script work by following this simple set of rules, depending on the edit operations:
• for word deletions: the predicted sequence is shorter, thus add a dummy token in the output sequence at the correct index to realign the sequences; • for word additions: the predicted sequence is longer, thus add a dummy token in the gold sequence at the correct index to realign the sequence; • for word substitutions: do nothing;
6https://universaldependencies.org/conll18/ evaluation.html
• The syntactic information of the inserted token must differ from that of the corresponding word in the other sequence. Thus every insertion and deletion are considered parsing errors.
Results: Speech recognition effect on parsing quality In Table 1, we observe that both graphbased and seq2label-based approaches give similar results when using no additional information, which shows that the limiting factor of the model is the speech recognition, rather than the parsing.
It is important to note that due to the nature of the speech corpus (spontaneous discussions), the WER is higher than what is typically expected on ASR benchmarks (usually containing ‘read’ speech). As a matter of fact, the ASR module used in our model reaches around 8 WER when trained and evaluated
on CommonVoice5.1 (Ardila et al., 2020). Further evidence of the limitation caused by the speech recognition module is shown in Table 3: changing the number of parameters of the graphbased parser does not significantly alter performance. Additionally, in Table 2 we observe a clear improvement in all the parsing metrics when evaluating on a test corpus with better speech recognition performance. The model’s speech recognition ability directly affects the number of predicted tokens (some words may be deleted or added), which in turn impacts parsing.
Results: Difference between sequence labeling approach and graph-based approach It is somewhat surprising that on the text modality (PIPELINE), the sequence labeling parser outperforms the graph-based approach, since this is not the case on the other modality (AUDIO). However, it does not outperform a larger graph-based model with an additional character-bi-LSTM such as hops. The character bi-LSTM may mitigate the impact of out-of-vocabulary words produced by misspelling errors from the ASR.
A hypothesis about the graph-based model per-
formance on AUDIO and the ORACLE settings may be that it is able to extract more relevant syntactic information from the signal due to its global decoding than simpler approaches such as sequence labeling.
The largest gap between the two parsing approaches occur when more information about speech segmentation is given to the models (ORACLE), reducing the overall influence of the speech recognition task on parsing.
Transcribe then parse or directly parse ? The PIPELINE approach with hops does reach a similar performance as the AUDIO model with our graphbased parser. However, hops is a more complex model not fully comparable to our graph-based parser. Moreover, it has 50% as many parameters as the model working directly on audio, requires 2 pretrained models, and is thus more expensive to train.
Lastly, Table 2 shows that the AUDIO approach outperforms the PIPELINE approach when the quality of the speech recognition improves. This result suggests that parsing benefits from AUDIO as soon as ASR reaches reasonable quality.

heading: 5 Conclusion
text: We introduced a graph-based speech parser that takes only the raw audio signal as input and assessed its performance in various settings and in several control experiments. We show that a simple graph-based approach with wav2vec2 audio features is on a par with or outmatches a more complex pipeline approach that requires two pretrained models.
From control experiments (ORACLE), we show that acquiring good quality word representations directly from speech is the main challenge for speech parsing. We will focus future work on improving the quality of word segmentation on the speech signal.
Limitations
We only evaluate our parsers on French, due to the availability of a large treebank, hence our conclusions should be interpreted with this restricted scope. We plan to extend to other languages and treebanks in future work.
We did not do a full grid search for hyperparameter tuning, due to computational resource limitations and environmental considerations, although we dedicated approximately the same computation
budget to each model in a dedicated setting. However, we acknowledge that not doing a full hyperparameter search may have affected the final performance of the parsers.

heading: Acknowledgements
text: This work is part of the PROPICTO project (French acronym standing for PRojection du langage Oral vers des unités PICTOgraphiques), funded by the Swiss National Science Foundation (N°197864) and the French National Research Agency (ANR20-CE93-0005). MC gratefully acknowledges the support of the French National Research Agency (grant ANR-23-CE23-0017-01).

heading: A Training Details
text: Table 4 and 5 describe in more detail the hyperparameters used for each parser for the different sets of modalities.

heading: Future Work
text: We will focus future work on improving the quality of word segmentation on the speech signal