Introducing Semantics into Speech Encoders
Year: 2023

Abstract:
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a taskagnostic unsupervised way of incorporating semantic information from LLMs into selfsupervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) FF1 score by over 2%. Our approach, which uses no ASR data, achieves similar performance as methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.

heading: 1 Introduction
text: Realizing artificial intelligence (AI) that can understand and respond to spoken language is a north star for many speech and natural language processing (NLP) researchers. A particularly effective framework for this is the encoder-decoder architecture, where an encoder represents input audio signals as high-dimensional embeddings and a decoder converts said embeddings to outputs for different downstream tasks. Benchmarks for such systems include spoken language understanding, where intent, named entities, or slot values are predicted from input utterances (Yang et al., 2021; Bastianelli et al., 2020; Shon et al., 2022), and spoken question answering, where the start and end frames of
∗Equal Contribution
an input audio passage answering an input audio question are predicted (Lin et al., 2022a).
A particularly notable setup of the encoderdecoder framework is the universal representation setup (Yang et al., 2021), where a shared selfsupervised speech encoder is pretrained upstream once and frozen for all downstream tasks, then a different lightweight decoder is fine-tuned on each downstream task. This setup is appealing for building speech systems as maintaining a separate large specialized model for every task is not computationally efficient. The universal representation setup has been widely adopted in other areas of research, such as computer vision (Goyal et al., 2019; Ericsson et al., 2021) and NLP (Rogers et al., 2020; Qiu et al., 2020), and production when there are many downstream tasks or domains (Molino et al., 2019). The current state-of-the-art speech encoders under this setup are W2V2 and HUBERT (Yang et al., 2021; Baevski et al., 2020; Hsu et al., 2021), which are transformer-based models trained with self-supervised learning (SSL) on raw audio and have achieved impressive performance on various tasks.
Recently, analytical works found SSL speech encoders capture primarily acoustic, not semantic, information (Pasad et al., 2021). Thus, researchers proposed end-to-end systems (Chung et al., 2020b; Kim et al., 2021; Qian et al., 2021; Le et al., 2022; Seo et al., 2022; Lin et al., 2022a) that introduce semantic information through large language models (LLMs), such as ROBERTA (Liu et al., 2019) or BART (Lewis et al., 2019), which are pretrained to capture language semantics (Clark et al., 2019). This is typically accomplished by the pipeline approach (Bastianelli et al., 2020), which passes audio input through the SSL speech encoder, then bridge module, then LLM. The bridge module converts speech encoder embedding outputs into LLM token inputs (Lugosch et al., 2019; Rao et al., 2021; Lin et al., 2022a; Seo et al., 2022).
11413
Unsupervised ASR models (ASR-U) (Liu et al., 2020b; Baevski et al., 2021; Liu et al., 2022) have also seen recent success. The state-of-the-art ASR-U model uses generative adversarial networks (GANs) (Goodfellow et al., 2020) to generate text transcription from input audio (Liu et al., 2022).
Current works combining SSL speech encoders and LLMs do not satisfy the universal representation framework, as they either (1) rely on ASR data on the downstream task, which is expensive to collect and maintain, (2) are not lightweight, requiring training the whole system end-to-end, or (3) are not general, as they do not consider a wide variety of downstream tasks (Lugosch et al., 2019; Rao et al., 2021; Lin et al., 2022a; Seo et al., 2022). Similarly, ASR-U was proposed for speech recognition and the focus is not improving SSL speech encoders (Baevski et al., 2021; Liu et al., 2022).
We propose introducing Semantics into Speech Encoders, SSE, a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. Concretely, SSE adopts the pipeline approach to obtain semantic embeddings, with an ASR-U bridge connector to extract information from LLMs. As ASR-U is inherently noisy, SSE introduces attention residual connection (He et al., 2016; Vaswani et al., 2017) between the speech encoder and LLM. SSE also efficiently aligns the LLM with the speech encoder through adapter modules (Houlsby et al., 2019). SSE improves W2V2 (Baevski et al., 2020) and HUBERT (Hsu et al., 2021) on 3 SLU tasks across 3 datasets, all under the universal representation setup. SSE also outperforms state-of-the art noASR method, DUAL (Lin et al., 2022a), in SQA.
While recent works use ASR-U to augment existing speech encoders with phoneme-level LLMs (Feng et al., 2022; Meng et al., 2022; Shi et al., 2022; Hsu et al., 2022), subword-level LLMs contain much more pertinent and measurable semantic information (Clark et al., 2019). Other works in SQA rely on clustering to assign audio frames to frequent subword tokens, but this requires heavy finetuning on the downstream task (Lin et al., 2022a).
To the best of our knowledge, we are the first to propose a task-agnostic SSL speech encoder which directly interfaces with subword-based LLMs, unblocking many other applications and future work in this domain. To this end, attention residual con-
nections and adapters are essential to successfully extracting semantic information from noisy intermediary transcriptions. We summarize our contributions below:
• We propose using ASR-U components to augment SSL speech encoders for generating subword tokens with semantic information.
• The augmented SSL speech encoders can be connected with powerful LLMs seamlessly and yields state-of-the-art performance under the universal representation setup.
• We show attention residual connections and adapters are essential to combining and aligning speech and text encoders.

heading: 2 Related Works

heading: 2.1 Self-Supervised Speech Encoders
text: SSL speech encoders (Liu et al., 2020a; Chung et al., 2020a; Ling and Liu, 2020; Liu et al., 2021, 2020c; Chung et al., 2019; Baevski et al., 2019; Schneider et al., 2019; Baevski et al., 2020; Hsu et al., 2021; Qian et al., 2022; Zhang et al., 2022) are trained to learn and reconstruct pooled clustered representations of input audio from the original audio. The intuition for this objective comes from linguistics, where speech can be broken down into phoneme groups, where different chunks of input audio represent different phoneme groups. W2V (Schneider et al., 2019) trains a convolutional neural network model to reconstruct the quantized cluster representations. W2V2 (Baevski et al., 2020) uses transformers and a discrete codebook quantization module. HUBERT (Hsu et al., 2021) improves W2V2 by disentangling the clustering and SSL objectives and using a BERT-style encoder (Devlin et al., 2018). The speech processing universal performance benchmark (SUPERB) (Yang et al., 2021; Lin et al., 2022b; Tsai et al., 2022) shows SSL speech encoders are the most effective method for solving multiple downstream tasks with minimal fine-tuning. A recent analytical work finds SSL speech encoders successfully encode acoustic information, but lack semantic information (Pasad et al., 2021). In response, CONTENTVEC (Qian et al., 2022) propose disentangling the speaker and semantic content of audio via an SSL objective. SPEECHLM (Zhang et al., 2022) propose training a multi-modal speech and text encoder.

heading: 2.2 Large Language Models
text: In contrast to speech encoders, pretrained LLMs are shown to capture rich semantic information (Clark et al., 2019). These methods optimize variants of the masked language modeling (MLM) objective to train a large transformer model. BERT (Devlin et al., 2018) uses MLM to learn a transformer encoder. ROBERTA (Liu et al., 2019) introduces dynamic masking and a larger text corpus. BART (Lewis et al., 2019) supports generative modeling and adds a denoising objective, making it less susceptible to noisy text inputs. LONGFORMER (Beltagy et al., 2020) is pretrained for long documents by increasing the document length limit during pretraining. LLMs have been successfully integrated with speech models for specific semantic tasks (Chung et al., 2020b; Kim et al., 2021; Qian et al., 2021; Le et al., 2022; Seo et al., 2022; Lin et al., 2022a), but not under the universal representation framework.

heading: 2.3 Task-Specific Speech Models
text: Task-specific SLU systems outperform generic SSL speech encoders typically by using a LLM. These systems rely on ASR data to reliably interface the LLM. LUGOSCH (Lugosch et al., 2019) trains a LSTM bridge module to convert audio features into phonemes then text. CTI’s (Seo et al., 2022) bridge module uses ASR logits to compute a weighted average of token embeddings. In addition to improving the bridge module, other works attempt to also distill LLM embeddings into speech representations (Chung et al., 2020b; Cha et al., 2021; Kim et al., 2021; Agrawal et al., 2022). For optimizing targeted metrics, researchers have also experimented with reinforcement learning (Rao et al., 2021). While combinations of these methods achieve impressive performance, they do not satisfy the universal representation setup.

heading: 2.4 Unsupervised ASR
text: Recent work show the viability of unsupervised speech recognition. W2V2-U (Baevski et al., 2021) accomplished this by running Principal Component Analysis (PCA), k-means clustering, and mean pooling to convert W2V2 (Baevski et al., 2020) features into phoneme-granularity features, then trains a GAN model to output phoneme text from the post-processed model (Baevski et al., 2021). The state-of-the-art method for phoneme-level unsupervised ASR is W2V2-U2.0 (Liu et al., 2022) which
directly trains a CNN to output phonemes from W2V2 features and uses a reconstruction loss to tie the input audio with corresponding generated text. Both methods use WFSTs to decode the phonemes into raw text. While there have been preliminary attempts (Feng et al., 2022; Meng et al., 2022) to use W2V2-U2.0 with phoneme language models1, we are the first to combine it with semantically-rich subword-based LLMs.

heading: 2.5 Adapters
text: Adapters are intermediary layers added to a large pretrained encoder. Adapter weights are learned during fine-tuning while the rest of the pretrained model is frozen. Adapters serve the dual purpose of efficient fine-tuning and preventing overfitting. First used by computer vision researchers (Rebuffi et al., 2017), adapters now enjoy much success in the natural language processing community by efficiently tuning LLMs (Houlsby et al., 2019). In particular, the multilingual speech translation community found that adapters can effectively align SSL speech encoders and LLMs for spoken translation tasks (Li et al., 2020; Le et al., 2021).

heading: 3 Proposed Method
text: We propose to introduce semantics into SSL speech encoders by using ASR-U to interface with LLMs. Section 3.2 describes how to use ASR-U to link a speech encoder with a LLM. Section 3.3 describes how to combine both acoustic and semantic information and deal with ASR transcriptions errors. Finally, Section 3.4 describes how to align LLMs with the speech encoder for downstream tasks.

heading: 3.1 Problem Setting
text: Following the universal representation framework (Yang et al., 2021), our model consists of a large speech encoder, E : X → Z , mapping input audio, X ∈ X , to embeddings, Z ∈ Z , and a light-weight task decoder, Dω : Z → Yω, mapping embeddings to downstream task outputs, Yω ∈ Yω. The speech encoder, E , is pretrained once, then shared on all downstream tasks. The task decoder, Dω, is fine-tuned on its respective task, ω ∈ Ω. During fine-tuning, the majority of model weights are frozen. This ensures the model can be efficiently stored and deployed.
During pretraining, the speech encoder is trained on unlabelled audio, X ∈ X , and unlabeled text,
1https://huggingface.co/voidful/phoneme_byt5
Tu ∈ Tu. During finetuning, the model is trained on the labelled downstream dataset, (X,Yω) ∈ X × Yω. Notice, costly labelled ASR data is not required during pretraining or finetuning.

heading: 3.2 Unsupervised Semantic Representation as a Bridge
text: To incorporate semantic information into SSL speech encoders, E : X → Z , we wish to leverage subword-based LLMs, M : S → Z , that capture language semantics (Devlin et al., 2018; Liu et al., 2019; Lewis et al., 2019; Beltagy et al., 2020). The major challenge is the mismatch of input spaces. Speech encoders take raw audio as input, X ∈ X . LLMs take subword tokens as input, S ∈ S. SSE uses W2V2-U2.0 (Liu et al., 2022) as a bridge module (Seo et al., 2022), B : Z → S , to convert speech encoder embedding output into LLM subword tokens in a pipelined approach, ESSE = E ◦ B ◦M.
Following W2V2-U2.0, the bridge module, B uses a GAN (Goodfellow et al., 2020) to output phoneme sequences, P ∈ P , conditioned on input audio, X ∈ X . The GAN does not directly predict subword-level transcriptions, because sub-
word barriers are not easily deducible from acoustic speech embeddings and requires implicitly learning phoneme-to-subword mappings. Instead, the bridge module, B, uses a Weighted Finite State Transducer (WFST), W : P → S, which is fed known phoneme-to-subword mappings, to map the generator outputs into subword tokens. The generator, G : Z → P , and the discriminator, C : P → [0, 1], are both convolutional neural networks (CNNs). The GAN model is trained on the same regularized GAN objective as in W2V2U2.0 (Liu et al., 2022).
The vanilla version of our final model is composed of (1) SSL speech encoder, E : X → Z pretrained on unlabelled audio data, (2) a CNN+WFST bridge module, B = W ◦ G : Z → S, trained on unlabelled text and audio data, and (3) a LLM, M : S → Z , pretrained on unlabelled text data. We also add an upsampling layer, U : Z → Z to make the sequence length of the LLM output match the speech encoder output, such that E and ESSE share the same output space.
We choose the 15th layer of the W2V2 (Baevski et al., 2020) as our speech encoder, as the last layers overfit the self-supervised training objective hence providing worse acoustic representations (Fan et al., 2020; Baevski et al., 2021; Pasad et al., 2021). We choose BART (Lewis et al., 2019) as our LLM, as it is trained to denoise noisy input subword tokens, and we expect the bridge module to introduce some noise. We call this version of our model SSE-BASE. A depiction can be found in Figure 1a.

heading: 3.3 Combining Semantics and Acoustics with Residual Attention
text: We hypothesize certain tasks may require more acoustic information than others. For instance, named entity recognition (NER) requires the model
to implicitly transcribe parts of the input speech, a primarily acoustic task. Since the pipelined model may suffer from transcription errors introduced by ASR-U, naively using the pipelined approach introduces an information bottleneck at the bridge module. Hence, we propose adding a residual connection (He et al., 2016) between SSE-BASE and the speech encoder, E .
This can be done in two ways: (1) upsampling semantic embeddings and concatenating with speech embeddings, Z = [ZE ||U(ZM)], or (2) using multihead attention (Vaswani et al., 2017) to merge the two embeddings, Z = [ZE ||MHA(ZE , ZM, ZM)], where ZE ∈ Z is the output of the W2V2L15 (Baevski et al., 2020) and ZM ∈ Z is the output of BART (Lewis et al.,
2019). The former is a simpler but more naive method. The latter is more effective as the attention layers are able to learn the alignment between speech and semantic embeddings. Notice, (2) introduces more learnable parameters to the finetuning-step, but we find the number of new parameters inconsequential compared to the size of the lightweight decoder.

heading: 3.4 Aligning Pretrained Text Model with Adapters
text: Inspired by works from speech translation (Li et al., 2020; Le et al., 2021), we hypothesize that the LLM can easily be adapted for speech tasks through the use of adapters. We adopt the general recipe for adapters, where an adapter (Houlsby et al., 2019), composed of a LayerNorm and 2-layer ReLU neural network, is added to the end of each feed forward layer in the LLM and finetuned on downstream tasks. This introduces additional parameters to finetuning, but we find the number of new parameters inconsequential compared to the size of the lightweight decoder. We call the model using both residual attention and adapters SSE-TUNE, and outline it in Figure 1b.

heading: 4 Experiments

heading: 4.1 Dataset
text: To show the effectiveness of introducing semantics into speech encoders, we evaluate 3 SLU tasks, intent classification (IC), slot filling (SF), and named entity recognition (NER), and SQA
task across 4 datasets: Fluent Speech Commands (FSC) (Lugosch et al., 2019), Spoken Language Understanding Resource Package (SLURP) (Bastianelli et al., 2020), Spoken Language Understanding Evaluation (SLUE) (Shon et al., 2022), and Natural Multi-speaker Spoken Question Answering (NMSQA) (Lin et al., 2022a), covering a wide variety of speakers, microphones, and environments

heading: 4.2 Encoder Setup and Baselines

heading: 4.2.1 Spoken Language Understanding
text: To show SSE improves SSL speech encoders, we augment two state-of-the art speech encoders under the universal representation setup: W2V2 and HUBERT. Following prior works that found intermediary layers of W2V2 contain better representations (Pasad et al., 2021; Baevski et al., 2021), we consider the 15th layer and the last layer of W2V2, named W2V2L15 and W2V2L24 respectively.
As mentioned in Section 3, we show 2 versions of our model, SSE-BASE and SSE-TUNE. The former uses the pipelined approach to connect W2V2L15 with BART (Lewis et al., 2019) with no additional modifications. The latter introduces an attention residual connection and learnable adapters to combine acoustics and semantics together and align the LLM with the speech encoder respectively. We either connect the residual connection to the output of W2V2L15, yielding SSE-TUNE (W2V2L15), or to the output of HUBERT, yielding SSE-TUNE (HUBERT).
To show the importance of using LLMs, we compare against 2 very recent approaches for improving SSL speech encoders without LLMs, SPEECHLM (Zhang et al., 2022) and CONTENTVEC (Qian et al., 2022). As HUBERT-BASE
was used as the base speech encoder by both baselines, we also provide results where SSE-TUNE is used to augment HUBERT-BASE.

heading: 4.2.2 Spoken Question Answering
text: To show the effectiveness of SSE, we compare it against DUAL (Lin et al., 2022a), the state-ofthe-art SQA model which does not use ASR data. While both SSE and DUAL obtain frame-level tokens from speech input, SSE uses ASR-U to obtain its tokens, whereas DUAL uses clustering. As a result, SSE’s output tokens exists in the LLM’s existing vocabulary, whereas DUAL’s output tokens does not. Hence, DUAL must retrain the LLM on its output tokens.
We compare DUAL to the closest analogous SSE model, which is SSE-BASE but with adapter layers, SSE-BASE (ADAP). Similar to DUAL, both methods modify the LLM weights. Unlike DUAL, SSEBASE (ADAP) is lightweight, tuning only around 10% of the total parameters. To produces framelevel predictions, we remove the upsampling layer from SSE-BASE (ADAP). We choose W2V2L15 as our speech model and BART as our LLM, as it is robust to ASR errors.
We also show a PIPELINE model, which trains a W2V2 model on ASR data and a LONGFORMER LLM on text-only question answering data. It is worth noting that since evaluation is based on the frame-level, SSL speech encoders are not a baseline since they operate at the audio level.

heading: 4.3 Decoder Setup
text: To satisfy the universal representation setup, we adopt lightweight SLU decoders from SUPERB (Yang et al., 2021). For IC, the decoder
is sum pooling followed by a multilayer perceptron classifier trained with cross entropy loss. For the SF and NER tasks, the decoder is recursive neural network (RNN) that transcribes input audio into text. The decoder identifies named entities or slot values by surrounding them with named special tokens and is trained with connectionist temporal classification loss. For SQA, we adopt the same decoder as DUAL (Lin et al., 2022a), which is a linear layer classifying each subword embedding as the start or end or neither of an answer span.

heading: 5 Results

heading: 5.1 Spoken Language Understanding

heading: 5.1.1 Improving SSL Speech Encoders
text: As seen in Table 2, SSE significantly improves the SLU performance of both W2V2 and HUBERT, confirming that including semantic information drastically improves existing SSL speech encoder performance. Specifically, SSE-TUNE (W2V2L15) improves W2V2L15 on all tasks. SSETUNE (HUBERT) improves HUBERT on 3 out of 4 tasks, and is the best performing model overall. Comparing SSE-TUNE with SSE-BASE shows residual attention and adapters effectively counteracts bridge module transcription errors.
The relative performance gain for IC is more than SF or NER. Unlike IC, both SF and NER require the speech encoder to transcribe identified audio snippets, and transcription is a primarily acoustic task. Hence SF and NER require less semantic information than IC. Nevertheless, combining both acoustic and semantic information, as done by SSE-TUNE, provides the most consistent performance improvement, since the skip connection can learn which type of information is more needed.

heading: 5.1.2 Importance of LLMs
text: As seen in Table 2, SSE-TUNE (HUBERT-BASE) outperforms alternative approaches augmenting speech encoders, SPEECHLM (HUBERT-BASE) and CONTENTVEC (HUBERT-BASE). Unlike these alternative approaches, SSE-TUNE incorporate information from LLMs, which we found to be very beneficial for capturing semantic information as they are carefully pretrained objectives on large amounts of unlabelled text data.
It is noteworthy that SSE-TUNE is a general framework which can augment any speech encoder of our choice, including SPEECHLM and CONTENTVEC. Similarly, SSE-TUNE can directly integrate new LLMs without costly pretraining. We leave incorporating such encoders into SSE-TUNE as future work.

heading: 5.2 Spoken Question Answering
text: As seen in Table 3, SSE outperforms recent unsupervised clustering-based approaches, DUAL. In contrast to DUAL’s HUBERT cluster tokens, SSE’s ASR-U tokens are better aligned with LLMs and share the same space. Thus, SSE can better utilizes pretrained LLMs. Furthermore, SSE does not require carefully tuning the number of HUBERT cluster counts, as the vocabulary size of the LLM is fixed and consistent with ASR-U.

heading: 5.3 Ablation Study

heading: 5.3.1 Choice of Language Model
text: We find subword-based LLMs contain more information than phoneme-based LLMs (Clark et al., 2019). We empirically verify this by replacing our subword-based LLM, BART (Lewis et al., 2019), with popular character-based LLM, ByT5 (Xue et al., 2022), and phoneme-based LLM, T5lephone (Hsu et al., 2022) in SSE-BASE. As
seen in Table 4, the subword-based LLM perform the best as each subword token is more semantically meaningful than a phoneme or character. We believe T5lephone outperforms the Byt5 as it has better robustness to ASR-U errors. Overall, subword-based LLMs are the best choice for embedding semantic information in transcribed text.

heading: 5.3.2 Residual Attention and Adapters
text: To more carefully analyze the affect of residual attention and adapters in SSE-TUNE, we run experiments on all SLU datasets with and without each component. We denote these two design choices as (ResAtt) and (Adap) respectively. As seen in Table 4, both components provide ample performance improvement over SSE-BASE.
We also try the naive residual connection approach described in Section 3.3 by directly concatenating the LLM upsampled semantic embeddings to the speech embeddings. We call this approach SSE-BASE (RES). This method is less effective than SSE-BASE (RESATT) as it does not learn how to align speech and semantic embeddings, but still improves SSE-BASE, further validating our hypothesis that merging acoustic and semantic information is beneficial.
As seen in parameter breakdown for the SSETUNE (W2V2L15) model in Table 1, the number of new learnable parameters introduced by (ResAtt) and (Adap) are unsubstantial compared to the size of the lightweight downstream decoder. Specifically, the downstream task decoder accounts for 9.60% of the total model parameters. SSETUNE introduces only 10.47% more parameters than SSE-BASE during fine-tuning and 0.91% to the total model parameter count, but often provides significant performance improvement.

heading: 5.4 Comparison with Supervised ASR Methods
text: To quantify the effect of transcription errors introduced by the bridge module, we compute the word error rate (WER) of the bridge connector in SSETUNE, and compare it against standard W2V2 supervised ASR models (Baevski et al., 2020) trained on 10 minutes, 100 hours, and 960 hours of labeled ASR data. Table 5 confirms that less noisy transcripts, transcripts with lower WER, correlates with better downstream performance. The unsupervised model, which uses 960 hours of unlabelled data, can reach similar WER as a supervised model trained on 100 hours of labelled data,
indicating the effectiveness of the bridge module. On SLURP and SLUE, the relative drop in WER (> 20%) is substantially more than the relative drop in downstream performance (< 5%), verifying SSE-TUNE’s tolerance to noisy transcriptions. The robustness to ASR errors come from our choice of LLM, BART, which is trained to handle noisy inputs, residual connection to acoustic embeddings, and LLM alignment with adapters.

heading: 5.5 Comparison to Specialized SLU Models
text: To better quantify the performance improvement introduced by SSE, we compare against 2 specialized SLU models that do not abide by the universal representation framework: Kaldi+HerMiT, which is a pipelined Kaldi ASR (Povey et al., 2011) and HerMiT NLU (Vanzo et al., 2019) model reported in the SLURP paper (Bastianelli et al., 2020), and CTI (Seo et al., 2022), which is an end-to-end pipelined W2V2 (Baevski et al., 2020) ASR and ROBERTA (Liu et al., 2019) NLU model. To the best of our knowledge, CTI is the state-of-the-art SLU model.
In addition to unlabelled text, unlabelled audio, and downstream data, both Kaldi+HerMiT and CTI require 40 hours of downstream SLURP ASR data (Bastianelli et al., 2020). Kaldi+HerMiT requires an additional 24,000 hours of ASR data (Povey et al., 2016). CTI requires an additional 960 hours of ASR data (Panayotov et al., 2015). Neither use lightweight fine-tuning. Thus, such specialized SLU models are less general, more expensive, and require much more data. As seen in Table 6, SSE helps bridge the gap between tailormade models and more practical SSL speech encoders. We believe ASR-U errors plays a major role in the remaining gap, as the ASR-supervised Kaldi+HerMiT and CTI models have WER of 16.20% and 16.67% respectively, compared to
SSE’s ASR-U bridge with a WER of 51.51%.

heading: 5.6 Error Analysis
text: To better understand the semantic information captured by SSE, we study predictions made by both HUBERT and SSE-TUNE (HUBERT) on SLURPIC’s test set. We find HUBERT errors are made primarily between intents within the same or similar domains (e.g. calendar_set vs calendar_query). The performance bottleneck lies with distinguishing finer-grained in-domain intents. Table 7 shows that SSE-TUNE is better at differentiating finergrained intents.
SSE-TUNE’s misclassifications come primarily from errors made by its ASR-U bridge component. As seen in Table 8, the ASR-U WER of incorrect predictions made by HUBERT is much lower than that of incorrect predictions made by SSETUNE. When ASR-U returns resonable transcriptions (typically <50% WER), SSE-TUNE can correctly classify inputs that HUBERT cannot. Hence, the effectiveness of SSE is tightly coupled with the effectiveness of ASR-U.

heading: 5.7 Representation Visualization
text: To better see the impact of including semantic representations, we visualize the pooled audio snippet embedding for intent classification on SLURP-IC using t-distributed stochastic neighbor embedding (t-SNE) (Van der Maaten and Hinton, 2008). We
denote the ground truth label of each audio snippet by the color of its pooled embedding. As seen in Figure 2, the clusters produced by semantic embeddings are more spread out and better separated than those produced by just acoustic speech embeddings, indicating that SSE introduces new semantic information that existing speech encoders lack.

heading: 6 Conclusion
text: We presented a compelling case for introducing semantics into SSL speech encoders and an effective method of doing so. Our approach boosts the performance of existing speech encoders on multiple SLU and SQA tasks and datasets. We provide reasoning for what tasks may benefit more or less from incorporating semantics. Furthermore, our approach is task agnostic and can augment any existing SSL speech encoder. With SSE-TUNE, we show merging acoustic and semantic information and effectively aligning LLMs to the speech encoder on downstream tasks can further boost performance with minimal parameter overhead. As it can generalize to many downstream tasks, SSE provides an important step towards AI that can understand and respond to spoken language.

heading: A Appendix
text: A.1 Acknowledgments
We thank Zoey Zhiyu Chen for checking experimental results and Jiatong Shi for providing helpful discussion. This work was done during an internship at Meta AI and was partially supported by NSF 1829071, 1937599, 2106859, 2200274, 2211557, 2119643, 2303037, DARPA #HR00112290103/HR0011260656, NASA, SRC, Okawa Foundation Grant, Amazon Research Awards, Cisco research grant, Picsart Gifts, Snapchat Gifts, and an NEC research award.
A.2 ASR-U Bridge Training Objective Details
We adopt the same unsupervised training scheme as W2V2-U2.0 (Liu et al., 2022). Specifically, we train the generator, G, on GAN loss, Lgan, a gradient penalty term, Lgp, for better convergence, a smoothness penalty term, Lsp, to encourage consecutive speech segments to generate the same phonemes, a phoneme diversity term, Lpd, to diverse phoneme usage in output transcripts by maximizing entropy, and a self-supervised reconstruction loss, Lss, to encourage the generated phonemes to match the input audio.
The reconstruction term uses a separate linear head to classify each speech embedding into 1 of 64 clusters, ζt, obtained from running k-means on the Mel-frequency cepstral coefficient (MFCC)
features of the input audio (Hsu et al., 2021; Liu et al., 2022). The final GAN training objective, minG maxC L, is summarized in Equation 1. The training procedure for the bridge module is outlined in Figure 3. Similar to W2V2-U2.0 (Baevski et al., 2021), SSE bridge models are trained on unlabelled audio and text from the Librispeech (Panayotov et al., 2015) dataset.
L = Lgan + λLgp + γLsp + ηLpd + δLss Lgan = E
Tu [logC(Tu)] + E X [log(1− C(G(X))]
Lgp = E X,Tu
µ∼U(0,1) µ′=1−µ
[(||∇C(µG(X) + µ′Tu|| − 1)2]
Lsp = ∑
(pt,pt+1)∈G(X) ||pt − pt+1||2
Lpd = 1 |B| ∑
S∈B −HG(G(S))
Lss = − ∑
t
logPG(ζt|X). (1)
A.3 Hyperparameter Settings
A.3.1 Speech Encoder We augment both W2V2 (Baevski et al., 2020) and HUBERT (Hsu et al., 2021) by introducing semantics. Specifically, we use the W2V2-Large LV-60 model and HUBERT-Large models, which are pretrained on just unlabelled audio and implemented with the fairseq library (Ott et al., 2019).
A.3.2 Large Language Model We use BART (Lewis et al., 2019) as our LLM since it is pretrained to handle noisy input. In our SLU experiments, we use BART-Base model, which has lower computational overhead. For our SQA experiments, we use BART-Large, since SQA is a more challenging task. Note, unlike baselines that train the whole LLM, SSE freezes all weights in its LLM except adapters optionally, hence SSE has lower overhead. All LLMs were implemented using the huggingface library (Wolf et al., 2019).
A.3.3 Residual Attention and Adapters We choose the residual attention layer to be the same dimension as our speech encoder, which is 1024 for both W2V2 (Baevski et al., 2020) and HUBERT (Hsu et al., 2021). We implement our general recipe for adapters using the adapter-transformers
package (Pfeiffer et al., 2020) and pyTorch (Paszke et al., 2019).
A.3.4 Bridge Connector We follow the same hyperparameter settings reported in the W2V2-U2.0 paper (Liu et al., 2022). Specifically, we use a 2-layer CNN with stride 3. The model is trained on unlabelled Librispeech960 (Panayotov et al., 2015) data for 100,000 epochs with a learning rate of 5e-5 and 3e-4 for the generator and discriminator respectively. Decoding is done using a WFST in the same way as W2V2U2.0 (Liu et al., 2022). Similar to W2V2-U2.0, we pre-process the Librispeech-960 by removing silences with an unsupervised model, but not during fine-tuning or testing. We believe such techniques could further improve performance, but leave it as future work. The regularized GAN loss function hyperparameters, as stated in Section A.2 are set to 1.0/1.5, 1.5/2.5, 0/3, and 0.3/0.5 for λ, γ, η, and δ respectively.
A.3.5 SLU Training Details As mentioned in Section 4.3, we use the standard decoders provided by SUPERB (Yang et al., 2021). We ran a grid search on 5 settings for learning rate on an exponential scale of 2 around the default settings from SUPERB (Yang et al., 2021) and found said default hyperparameters optimal. Specifically, we set the learning rate to 1e− 4, 1e− 4, 2e− 4, and 2e− 4 for FSC-IC, SLURP-IC, SLURP-SF, and SLUE-NER respectively. All methods use the AdamW (Loshchilov and Hutter, 2017) optimizer with gradient clipping set to 1 for 200,000 total steps to convergence. Validation performance is used to pick the best model for all datasets except SLUE, since SLUE test data is not publicly available.
A.3.6 SQA Training Details As mentioned in Section 4.3, we use a frame-level linear layer classification head as our decoder. We follow DUAL’s (Lin et al., 2022a) default hyperparameter settings with a learning rate of 1e-4. We train the models using the same warm-up and decay strategies as DUAL with the AdamW (Loshchilov and Hutter, 2017) optimizer for 5,000 steps to convergence.
A.4 Training Setup and Time
All models were trained on a server with 8 Nvidia Tesla V100 GPUs. The total training time for the
bridge module takes around a day. The total training time for downstream tasks take between half a day and one day.
A.5 Dataset Details
As mentioned in Section 4.1, we evaluate SSE on 3 SLU tasks, intent classification (IC), slot filling (SF), and named entity recognition (NER), and the SQA task. The goal of IC is to classify the intent of an input audio snippet. The goal of SF is to extract certain attributes of a given intent from an audio snippet. The goal of NER is to identify named entities in an audio snippet. The goal of SQA is to find the start and end frames of the answer in a spoken passage given a spoken question.
A.5.1 FSC The FSC dataset (Lugosch et al., 2019) is an IC dataset for a smart home virtual assistant. The input is a single audio file containing spoken English commands and the output class is the intent of the spoken command. The data was obtained through crowd-sourcing from 97 native and non-native English speakers. In total, there are 31 intents. The number of utterances and hours of each split can be found in the Table 9.
A.5.2 SLURP The SLURP dataset (Bastianelli et al., 2020) is an IC and SF dataset for an in-home personal robot assistant. The input is a single audio file containing spoken English commands and the output is the scenerio, action, and entities. In total, there are 18 different scenarios, 46 different actions (IC), and 56 different entities (SF). The data was collected from 177 native and non-native English speaking Amazon Mechanical Turk workers. The number of utterances and hours of each split can be found in Table 9. SLURP use both headsets and microphones with various placement configurations.
A.5.3 SLUE The SLUE dataset (Shon et al., 2022) is a NER dataset using European Parliament event recordings. The input is a single audio file containing spoken English passages and the output are the named entities. There are in total 7 categories that were based on the OntoNotes Release 5.0 (Hovy et al., 2006) entity labels. The dataset was collected from the official European Parliament website. The
number of utterances and hours of each split can be found in the Table 9.
A.5.4 NMSQA The NMSQA dataset (Lin et al., 2022a) is a SQA dataset generated from a standard text question answering dataset, SQUAD-V1.1 2, using Amazon Polly Text-to-Speech 3 for the train and dev split, and 60 human speakers for the test set. NMSQA contains 297.18 hours, 37.61 hours, and 2.67 hours of train, dev, and test split audio respectively. We follow DUAL (Lin et al., 2022a) by evaluating on Frame-level F1 score (FF1) and Audio Overlapping Score (AOS).
2A question answering dataset using Wikipedia articles 3https://aws.amazon.com/tw/polly
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We leave incorporating such encoders into SSE-TUNE as future work