SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices

Authors: dongqi  cai
shangguang  wang
zeling  zhang
felix xiaozhu lin
mengwei  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 09:46:50

Abstract: Speech serves as a ubiquitous input interface for embedded mobile devices. 
Cloud-based solutions, while offering powerful speech understanding services, raise significant concerns regarding user privacy. 
To address this, disentanglement-based encoders have been proposed to remove sensitive information from speech signals without compromising the speech understanding functionality. 
However, these encoders demand high memory usage and computation complexity, making them impractical for resource-constrained wimpy devices.
Our solution is based on a key observation that speech understanding hinges on long-term dependency knowledge of the entire utterance, in contrast to privacy-sensitive elements that are short-term dependent. 
Exploiting this observation, we propose SILENCE, a lightweight system that selectively obscuring short-term details, without damaging the long-term dependent speech understanding performance.
The crucial part of SILENCE is a differential mask generator derived from interpretable learning to 
automatically configure the masking process.
We have implemented SILENCE on the STM32H7 microcontroller and evaluate its efficacy under different attacking scenarios. 
Our results demonstrate that SILENCE offers speech understanding performance and privacy protection capacity comparable to existing encoders, while achieving up to 53.3$\times$ speedup and 134.1$\times$ reduction in memory footprint.

Main paper: 
 introduction: 
 Privacy concern for cloud speech service The volume of speech data uploaded to the cloud for spoken language understanding (SLU) is steadily increasing [1, 13, 2] , particularly in ubiquitous wimpy devices where textual input is inconvenient [50, 19, 3 ], e.g., home automation devices [40] , smartwatches [46] , telehealth sensors [26] and smart factory sensors [35] . However, exposing raw speech signal to the cloud raises privacy concerns [51] . It was revealed that contractors regularly listened to confidential details in Siri recordings to improve its accuracy [4] . This included private discussions, medical information, and even intimate moments.There are many aspects of potential privacy leakage in cloud-based SLU. Among them: biometric or contextual privacy leakage have been well studied and somewhat solved by removing information relevant to such tasks without compromising the SLU accuracy [20, 43] ; transcript protection (especially sensitive entities) is more challenging since it is deeply entangled with the SLU task itself. As shown in Figure 1 , this paper focus on ensuring that cloud-based systems could efficiently classify the intent of SLU task (e.g., scheduling appointments or controlling home devices) while refraining from identifying the concrete entities (e.g., unintended names or passwords) in the spoken utterance, i.e., high word error rate (WER) of Automatic Speech Recognition (ASR) task. This is also a setting commonly used in speech privacy protection [55, 11, 17, 51, 16] .Prior approaches A prevalent method for private speech processing is employing encoders1 based on disentanglement representation learning [55, 11, 34, 42] , as illustrated in Figure 1 (b). Those encoders extract the speech representations using pre-trained acoustic models, e.g., wav2vec [49, 11] , conformer [30, 42] and Preformer [23, 55] . Furthermore, they promote representation disentanglement through adversarial training [29] . For example, PPSLU [55] uses a 12-layer transformer-based Preformer as its encoder.As a result, disentanglement-based encoders still demand considerable computational resources, often exceeding tens of GFLOPs, to achieve effective disentanglement [12] . They are also memoryintensive, often comprising tens of millions of parameters. Consequently, they are unsuitable for embedded devices with limited memory. Moreover, it takes time-consuming adversarial training to disentangle the encoded representation for each specific SLU task. This aspect limits the flexibility and scalability for emerging SLU tasks. More motivating details will be presented in §2.2.In this paper, we aim to achieve the real-time, privacy-preserving offloading of speech understanding task on wimpy devices like STM32H7 microcontroller [5] with only 1MB RAM. This goal necessitates a novel encoder design that must be both lightweight and effective in filtering out sensitive information, as illustrated in Figure 1(c) .Our solution We therefore present SILENCE, a SImpLe ENCodEr designed for efficient privacypreserving SLU offloading. It is based on the asymmetric dependency observation: SLU intent extraction (e.g., scenario identification) typically requires only long-term dependency knowledge across the entire utterance, while ASR task (e.g., recognizing individual words or phrases) needs short-term dependency, as confirmed by our experiments in §3.1. Based on it, SILENCE strategically partitions the utterance into several segments, selectively masking out the majority to enhance privacy by obscuring short-term details, without significantly damaging the long-term dependencies. The processed audio waveform is then transmitted to the cloud for SLU intent analysis. Additionally, we integrate a differential mask generator, inspired by interpretable learning methods [21] , to optimize performance by automatically identifying how many and which segments to mask.We deploy SILENCE on the STM32H7 microcontroller [5] and assess its performance using the SLURP dataset [14] in both black-box and white-box attack environments. SILENCE achieves 81.2% intent classification accuracy on SLURP, surpassing previous privacy-preserving SLU systems by up to 8.3%. Regarding privacy protection, SILENCE offers comparable security to earlier systems, with a word error rate of up to 81.6% and an entity error rate of 90.7% under malicious ASR attacks. Even against white-box attacks, where attackers are strongly assumed to have the same encoder structure and weights as SILENCE, plus partial data from malicious clients, SILENCE maintains 67.3% word error rate and 64.3% entity error rate. Additionally, SILENCE proves to be resource-efficient and feasible for wimpy devices, using only 394.9KB of memory and taking just 912.0ms to encode a 4-second speech signal. Integrated with RPI-4B for a fair comparison, SILENCE uses up to 134.1× less memory and operates up to 53.3× faster than prior systems. The accuracy of SILENCE is only 7% lower than unprotected SLU systems.We have made the following contributions.• Based on the observation of asymmetric dependency between SLU and ASR tasks, we propose SILENCE, a simple yet effective encoder system for privacy-preserving SLU offloading.• We are the first to retrofit interpretable learning methods to automatically configure the masking process for a better balance between privacy and utility in speech understanding tasks.• We evaluate SILENCE on a wimpy microcontroller unit and demonstrate its effectiveness under various attack scenarios.2 Related Work and Background 
privacy-preserving slu: 
 Spoken Language Understanding (SLU) is a critical component of modern voice-activated systems, responsible for interpreting human speech and translating it into structured, actionable commands. For instance, when a user says, "Set a meeting for tomorrow at 10 AM," the SLU system might map this to a structured intent such as {scenario: Calendar, action: Create_entry}. Longdependent intend classification is currently the main objective of SLU understanding literature and has a wide range of application scenarios [54, 24, 52, 9, 44, 14, 57, 18, 22] .The evolution of SLU systems has seen a shift from traditional twocomponent systems, comprising ASR and Natural Language Understanding (NLU), to modern endto-end neural networks [48, 31] . These advanced systems bypass the intermediate textual representation and directly map speech signals to their semantic meaning, enhancing efficiency and reducing error propagation. A typical end-to-end SLU model features an encoder, often with convolution and attention-based elements, and a decoder, including a transformer decoder and a connectionist temporal classification decoder. Many SLU systems incorporate encoders from pre-trained ASR models like HuBERT [56] , replacing the original ASR decoder with one tailored for SLU tasks.Threat Model Our threat model aligns with prior work [55, 11] where users (the victims) actively offloads their audio data to the cloud server (the adversary) for intended SLU tasks. Upon receiving the data, the adversary may employ automatic speech recognition to transcribe the audio and identify private entities [17, 51, 16] . Note that the transcriptions are often exceedingly detailed, containing much more information than the users intend to disclose. The goal of this paper is to ensure that the victims can reliably obtain the predefined SLU intent from the adversary, while preserving the adversary from discerning sensitive details or private entities in the transcript.For instance, home pods might capture recordings of confidential daily interactions alongside explicit commands, presenting a paradigmatic case for SILENCE. Without SILENCE, over 80% of our private daily conversations could be automatically recognized and stored for unforeseen usage as will be analyzed in §5.1.Privacy-preserving methods Crypto-based approaches, such as HE [60] and MPC [28] , have been proposed to provide encrypted computation. Unfortunately, they are technically slow and thus impractical for deployment on resource-constrained audio devices due to the significant increase in computation and communication complexity. For example, MPC-based PUMA [25] takes 5 min-utes to complete one token inference, which is far too slow for real-time. Voice conversion is another method to protect speech content. Prεεch [10] integrates voice conversion with GPT-based generated noise protect privacy, but it is far from feasible for deployment on wimpy devices. Traditional peripheral devices, such as ultrasonic microphone jammers (UMJ), are designed to obscure raw speech by inserting non-linearity noise, thereby preventing illegal eavesdropping [27, 16] ; however, they also corrupt speech semantics as well. A emerging and prevailing strategy is disentanglingbased encoders [11, 55, 34] ; they aim to create a disentangled and hierarchical representation of the speech signal devoid of sensitive data. But we reveal their performance issue next. [55] for a 4-second audio inference. We conduct preliminary experiments to measure the resource consumption of the disentanglingbased encoder of a pre-trained SLU model on a Raspberry Pi 4B (RPI-4B) [6] and Jetson TX2 (TX2) [7] . Our key observation is that disentangling-based privacy-preserving SLU system is too resource-intensive for practical deployment. As illustrated in Figure 2 , a disentanglement encoder consumes 648.7MB memory and 12.8s for complete one inference on RPI-4B. Even in the strong TX2 with GPU, the encoder still takes 593.0ms to complete one inference. Considering the network latency, the end-to-end latency of the disentangling-based SLU offloading system only saves 0.7% wall-clock time compared to the OnDevice inference without offloading, with a similar memory footprint over 500M.Implications Disentangling-based encoders is slow and memory-intensive due to the complex encoder structure designed to separate sensitive information from the speech signal. Given the limited resource of wimpy devices, it is not practical for common privacy-preserving SLU scenarios. To enable practical privacy-preserving SLU, the encoder structure and the inference process need to be simplified. 
system design and rationales: 
 We introduce SILENCE to efficiently scrub raw audio for privacy-preserving SLU, as depicted in Figure 3 . The key idea of SILENCE is simple and novel: it masks out a portion of audio segments before sending them to the cloud for SLU tasks. This design is based on an unique observation shown in Figure 4 (c): when a portion of audio segments is masked out, the ASR model becomes incapable to recognize the phonemes in the masked frames, while the SLU model can still recognize the intent. Design rationale Why is SILENCE able to protect the sensitive entity privacy while maintaining SLU accuracy? This capability is rooted in the asymmetrical dependency between the ASR and SLU task.Speech is composed of many meta phonemes, and the generation of a single meta phoneme depends on its adjacent frame [51] . Dependency is defined as the length of frame that a model's output depends on. Figure 4 (a) shows each phoneme is mainly dependent on a few frames, indicating shortterm dependency. This phenomenon is referred to as "peaky behavior" in the ASR literature [59] . In contrast, an SLU model utilizes an attention-based decoder [56] to capture the relationship between the entire utterance and the intent, implying that the intent is long-term dependent on the whole utterance.Formally, SILENCE is a simple encoder based on asymmetrical dependency-based masking. This simple masking encoder is defined as: x = x ⊙ Z, where x is the input audio signal, ⊙ represents the element-wise multiplication, x is the masked audio signal and Z is the binary masking vector with the same dimension as x. Z consists of k uniform portion, with all 0s or 1s in one portion to mask-out or preserve the complete adjacent frames, respectively. This simple encoder forms the basis of SILENCE's efficiency and privacy-preservation capacity, enabling secure offloading of speech understanding tasks on wimpy devices.The configuration challenges: Figure 4 (c) demonstrates that the ratio of masked portion plays a crucial role in balancing the privacy (WER-ASR) and utility (ACC-SLU). Currently, SILENCE employs a trivial masking mechanism, necessitating clients to undertake a time-intensive hyperparameter adjustment about the extent and location of masking. Incorrect masking configurations can result in significant loss of global long-term dependency, negatively affecting SLU accuracy, or insufficient masking of sensitive information, thus compromising privacy. Therefore, we face critical questions: how many and which portions should be masked?To address these challenges, we derive a differential mask generator from the interpretable learning [21] as a online configurator for SILENCE. This automatically generate the masking vector Z.The mask generator is trained to identify how many and which portions to mask, optimizing the privacy-utility balance.Differentiable mask generator The configurator model aims to minimize the discrepancy between masked and original output by generating a mask Z. Formally, we define the number of unmasked portions as L 0 loss:L 0 (ϕ, x) = n i=1 1 [R ̸ =0 ] (Z i ) (1)where ϕ is the mask generator, 1(•) is the indicator function. We minimize L 0 for dataset D, ensuring that predictions from masked inputs resemble those from the origin model: EQUATIONEQUATIONwhere ŷ = f (x), y is the tokenized label, D ⋆ [y∥ŷ] is the KL divergence and the margin γ ∈ R >0 is a hyperparameter.Given that L 0 is discontinuous and has zero derivative almost everywhere, and the mask generator ϕ requires a discontinuous output activation (like a step function) for binary masks, we utilize a sparse relaxation to binary variables [36, 15] instead of the binary mask during training.Holistic workflow As shown in Figure 5 , SILENCE encompasses two phases:(1) Offline phase: (1a) First, SILENCE trains a differentiable mask generator. The client selects a mask generator model, potentially a submodule of a pre-trained ASR model, such as HuBERT's CNN feature extractor. A small gate model is then integrated with this submodule. The combined model processes the input audio and generates a mask. This mask selectively conceals parts of the input, ensuring retention of only vital SLU information while hiding sensitive data. The masked input is then forwarded to either a trusted cloud service or a local SLU model for obtaining masked output. The mask generator is fine-tuned to minimize the discrepancy between the masked output logits and the original intent, as defined in Equation (1-3).(1b) Second, SILENCE adapts the cloud model . Here, the client forwards the masked input and a specific SLU intent (e.g., "set alarm") to the cloud-based SLU model. The model undergoes finetuning to adapt to the masked inputs. This process includes adjusting the model parameters for accurate recognition and response to SLU commands based on the masked input.(2) Online phase: In online speech understanding, the client sends the masked input to the cloud SLU model. Using the adapted model, the cloud-based SLU accurately identifies and executes the intended SLU action or response.Training the differentiable mask generator is affordable for the client.Our experiments indicate that convergence is achieved with approximately 200 audio samples, equivalent to 600 seconds of audio. This process takes up to 30 seconds on an A40 GPU. Adapting the SLU model to each mask generator is a one-pass effort. This adaptation is relatively trivial, especially when starting from a fine-tuned SLU model rather than building from scratch. This aspect of the process incurs minimal cost compared to the training of the cloud SLU model. Moreover, these costs can be amortized over a large number of edge users in the long run, making it an economically viable solution.Remark Note that the mask generator is not developed for tagging sequences at a semantic level. Rather, its design focuses on identifying segments that are more relevant to the SLU task. This task is essentially a relatively straightforward binary classification problem, which is proven to be effective in prior interpretable learning literature [21, 15] and light-weight enough for real-time inference. 
implementation and methodology: 
 We have fully implemented the SILENCE prototype atop SpeechBrain [47] , a PyTorch-based and unified speech toolkit. As prior work [56] , we use SpeechBrain to train the differential mask generator and simulate the cloud training process. After that, we deploy the trained mask generator into the embedded devices and evaluate the end-to-end performance.Hardware and environment Offline training is simulated on a server with 8 NVIDIA A40 GPUs.The trained mask generator is deployed into the STM32H7 [5] or Raspberry PI 4 (RPI-4B) [6]. STM32H7 is a resource-constrained microcontroller with 1MB RAM. RPI-4B is a popular development board with 4GB RAM. We embed the approaches not feasible to fit in the STM32H7 into the RPI-4B.We design four types of mask generator structures: (1) Random: a random binary vector generator with 50% portion masked; (2) SILENCE-S: a learnable mask generator with only one MLP gate;(3) SILENCE-M: a learnable mask generator with one HuBERT encoder layer and the gate; (4) SILENCE-L: a learnable mask generator with three HuBERT encoder layers and the gate. As for the cloud SLU model, we simulate it using the SoTA end-to-end SLU model [56] . It replaces the ASR decoder of pre-trained HuBERT with SLU attentional decoder.We run our experiments on SLURP [14] and FSC [37] . FSC is a widely used dataset for spoken language understanding research. SLURP's utterances are complex and closer to daily human speech, We select scenario classification accuracy to measure the SLU understanding performance (ACC-SLU). Following prior work [55] , we choose large-scale English reading corpus LibriSpeech [41] for a multi-task protection scenario. In the multi-task protection scenario, not only the SLU command utterance (SLURP/FSC) but also the background or the subsequent utterance (LibriSpeech) are uploaded to the cloud. WER is used to measure the attack performance. More specifically, we utilize WER-SLU to measure the attacker's capacity to recognize the word information in the uploaded SLU audio itself, and WER-ASR as the WER of recognized accompanying audio, i.e., LibriSpeech dataset. We also report the private entity recognition error rate (EER) to ensure that the cloud model is not able to recognize the private information in the speech signal. As for latency, we sequentially fed test audios into the local model without any window processing2 and recorded the average forward time as the local execution time.Baselines We compare SILENCE to the following alternatives: (1) OnDevice means the cloud SLU model is downloaded and run locally on the client device.(2) AllOffload means the raw audio is uploaded to the cloud for SLU inference. (3) VAE [11] is the vanilla variational auto-encoder method that uses adversarial training to disentangle the private information from speech signal. (4) PPSLU [55] is the state-of-the-art disentangling-based SLU privacy-preserving system, which uses 12 transformer layers to separate the SLU information into a part of the hidden layer and only sends those hidden layers to the cloud for SLU inference. Attack scenarios. As illustrated in Figrue 6, we use five attacks encompassing both active and passive attacks: (1) Azure represents a passive black-box attacker scenario, in which the masked audio is transmitted to Azure [38] for automatic speech recognition. (2) Whisper simulates a SoTA cloudbased ASR model. This passive black-box attacker uses the pre-trained W hisper.medium.en model [45] , directly downloaded from HuggingFace [58] . (3) Whisper(White-box) constitutes an active white-box attack. Here, we hypothesize that certain users are malicious and disclose the mask generator's structure and weights, along with their own audio data, to the Whisper attack model. Whisper(White-box) then utilizes this collected data from malicious users to adapt the pre-trained W hisper.medium.en model to the specific masking pattern. (4) U-Net is a traditional inpainting model based on convolutional U-Net structure, commonly used in literature to actively reconstruct missing audio signals [32, 33] . We utilize the SLURP training set and their masked counterparts to train the inpainting model from scratch to reconstruct the missing audio. (5) CQT-Diff is a neural diffusion model with an invertible Constant-Q Transform to leverage pitch-equivariant symmetries [39] , allowing it to effectively reconstruct audio without retraining.Hyper-parameters During the offline phase in Figure 5 , we use the Adam optimizer with a learning rate of 1e-5 and a batch size of 4. For the inference step, we use the batch size of 1 to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [53] . KL threshold λ is set as 0.15 for all mask generators. Attack model is set as Whisper without special declaration. We have an illustrative example of the generated masks on audios selected randomly from SLURP in Appendix A. 
evaluation: 
 5.1 End-to-end performance SILENCE achieves comparable accuracy performance and privacy protection capacity to previous encoders. As shown in Figure 7 , we compare the accuracy of SILENCE with all baselines. OnDevice offloads no signals to the cloud and thus has the best privacy protection (WER=100). It is observed that SILENCE could achieve up to 81.1% accuracy, with less than 7% accuracy loss compared to unprotected AllOffload and local OnDevice SLU model. Its rationale is that we mainly mask the short-dependent frames that does not significantly affect the SLU performance. We also compare the performance of SILENCE with the SoTA privacy-preserving SLU system, i.e., PPSLU [55] . SILENCE achieves 7.2% higher accuracy than PPSLU which tries to apply complex nonlinear transformation to the hidden layer to prevent malicious re-construction, but this might also damage part of the SLU information. In terms of privacy preservation, our learnable mask generator achieves up to 78.6% WER using SILENCE-L, indicating a privacy-preserving capacity on par with PPSLU. The same benefits exist in FSC dataset as well. SILENCE demonstrates more than 99% intent understanding accuracy, similar to all the baselines, while effectively defends against sensitive word recognition attacks, achieving more than 80% WER, outperforming all disentanglement-based protections. Furthermore, we complete the inference with much lower delays and memory footprint as will be shown in Figure 10 . W hisper.medium.en model to the specific masking pattern.Hyper-parameters During the offline phase in Figure 5 , we use the Adam optimizer with a learning rate of 1e-5 and a batch size of 4. For the inference step, we use the batch size of 1 to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [42] . KL threshold l is set as 0.15 for all mask generators. Attack model is set as Whisper without special declaration.In this section, we evaluate the performance of SILENCE under different attack scenarios, and also assess the effect of varying hyper-parameters on the privacy-utility trade-off. After that, we discuss the system cost of SILENCE.5.1 End-to-end performance SILENCE achieves comparable accuracy performance and privacy protection capacity to previous encoders. As shown in Figure 6 , we compare the accuracy of SILENCE with all baselines. Is observed that SILENCE could achieve up to 80% accuracy, with less than 8% accuracy loss compared to SILENCE is resistant to different attack models. As illustrated in Table 1 , SILENCE increases the SLU-WER from 14.7% to 78.6% under the attack model Whisper. As for the online attack model Azure, SILENCE increases the SLU-WER from 14.7% to 81.6%. According to our returned service details, we find that over 50% of the sent audios are tagged as "ResultReason.NoMatch", which means audios are recognized as null utterances by the Azure ASR model. Fine-tuned Whisper is a white-box attack model, which means the attacker has the same mask generator structure and weights as the SILENCE. We still achieve more than 50% SLU-WER under this attack model. This is because even Fine-tuned Whisper is fine-tuned to fill some of the missing frames, it still could not recover the private missing frames because ,masking the short-dependent frames have fundamentally destroy the raw audio signal. It is not possible to re-construct the phoneme without knowing any speech information. In the last row, we show the high entity error rate to demonstrate that the private entity is not leaked. W hisper.medium.en model to the specific masking pattern.Hyper-parameters During the offline phase in Figure 5 , we use the Adam optimizer with a learning rate of 1e-5 and a batch size of 4. For the inference step, we use the batch size of 1 to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [42] . KL threshold l is set as 0.15 for all mask generators. Attack model is set as Whisper without special declaration.In this section, we evaluate the performance of SILENCE under different attack scenarios, and also assess the effect of varying hyper-parameters on the privacy-utility trade-off. After that, we discuss the system cost of SILENCE. 1 , SILENCE increases the SLU-WER from 14.7% to 78.6% under the attack model Whisper. As for the online attack model Azure, SILENCE increases the SLU-WER from 14.7% to 81.6%. According to our returned service details, we find that over 50% of the sent audios are tagged as "ResultReason.NoMatch", which means audios are recognized as null utterances by the Azure ASR model. Fine-tuned Whisper is a white-box attack model, which means the attacker has the same mask generator structure and weights as the SILENCE. We still achieve more than 50% SLU-WER under this attack model. This is because even Fine-tuned Whisper is fine-tuned to fill some of the missing frames, it still could not recover the private missing frames because ,masking the short-dependent frames have fundamentally destroy the raw audio signal. It is not possible to W hisper.medium.en model to the specific masking pattern.Hyper-parameters During the offline phase in Figure 5 , we use the Adam optimizer with a learning rate of 1e-5 and a batch size of 4. For the inference step, we use the batch size of 1 to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [42] . KL threshold l is set as 0.15 for all mask generators. Attack model is set as Whisper without special declaration.In this section, we evaluate the performance of SILENCE under different attack scenarios, and also assess the effect of varying hyper-parameters on the privacy-utility trade-off. After that, we discuss the system cost of SILENCE. 1 , SILENCE increases the SLU-WER from 14.7% to 78.6% under the attack model Whisper. As for the online attack model Azure, SILENCE increases the SLU-WER from 14.7% to 81.6%. According to our returned service details, we find that over 50% of the sent audios are tagged as "ResultReason.NoMatch", which means audios are recognized as null utterances by the Azure ASR model. Fine-tuned Whisper is a white-box attack model, which means the attacker has the same mask generator structure and weights as the SILENCE. We still achieve more than 50% SLU-WER under this attack model. This is because even Fine-tuned Whisper is fine-tuned to fill some of the miss- W hisper.medium.en model to the specific masking pattern.Hyper-parameters During the offline phase in Figure 5 , we use the Adam optimizer with a learning rate of 1e-5 and a batch size of 4. For the inference step, we use the batch size of 1 to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [42] . KL threshold l is set as 0.15 for all mask generators. Attack model is set as Whisper without special declaration.In this section, we evaluate the performance of SILENCE under different attack scenarios, and also assess the effect of varying hyper-parameters on the privacy-utility trade-off. After that, W hisper.medium.en model to the specific masking pattern.Hyper-parameters During the offline phase in Figure 5 , we use the Adam optimizer with a learning rate of 1e-5 and a batch size of 4. For the inference step, we use the batch size of 1 to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [42] . KL threshold l is set as 0.15 for all mask generators. Attack model is set as Whisper without special declaration.In this section, we evaluate the performance of SILENCE under different attack scenarios, and also assess the effect of varying hyper-parameters on the privacy-utility trade-off. After that, SILENCE is resistant to different attack models. As illustrated in Figure 8 , SILENCE increases the SLU-WER from 14.7% to 78.6% under the attack model Whisper. As for the online attack model Azure, SILENCE increases the SLU-WER from 14.7% to 81.6%. According to our returned service details, we find that over 50% of the sent audios are tagged as "ResultReason.N oM atch", which means audios are recognized as null utterances by the Azure ASR model. Whisper(White-box) is a white-box attack model, which means the attacker has the same mask generator structure and weights as the SILENCE. We still achieve more than 50% SLU-WER under this attack model. This is because even Whisper(White-box) is fine-tuned to fill some of the missing frames, it still could not recover the private missing frames. Because masking the short-dependent frames fundamentally destroys the raw audio signal. It is not possible to re-construct the phoneme without knowing any speech information. In the last subfigure, we show the high entity error rate to demonstrate that the private entity is not leaked. SILENCE scales to better privacy-accuracy trade-off with a larger mask generator. We explore the impact of the threshold γ of SILENCE under different mask generator structures. As shown in Figure 9 , the threshold γ controls the trade-off between the privacy and utility. When γ is small, the mask generator is more conservative, leading to higher the utility a lower the masking portion.As we have discussed in Section 3, a lower rate of masking portions leads to higher possibility of privacy entity leakage. When γ is large, the mask generator is more aggressive, enhancing privacy. Another way to achieve more practical privacy-utility balance is using a more complex mask generator structure, e.g., SILENCE-L. It achieves higher utility with the same privacy level compared to SILENCE-S, albeit with less efficiency, as shown in § 5.2.SILENCE protects the private entities efficiently as shown in Figure 10 . Different from prior encoders using complex disentanglement model, SILENCE only requires a light-weight mask generator to scrub the private information. The size of this generator varies according to different mask generator structures. For the smallest mask generator, SILENCE-S, it only requires a 394.9KB memory footprint, and could successfully embed into the resource-constrained STM32H7 with 2MB RAM. SILENCE is efficient not only in terms of memory footprint but also in latency. SILENCE-S completes the local encoding with only 912.2ms on the resource-constrained STM32H7. For a fair comparison, we embed SILENCE-S into RPI-4B and find that it is 18.1× faster and 134.1× less memory footprint than PPSLU. Even with the strong mask generator SILENCE-L, SILENCE achieves up to 7.5× lower encoding latency and consumes 1.9× less memory compared to OnDevice. 
conclusion and discussion: 
 SILENCE is an efficient and privacy-preserving end-to-end SLU system based on the asymmetrical dependency between ASR and SLU. SILENCE selectively mask the short-dependent sensitive words while retaining the long-dependent SLU intents. Together with the differentiable mask generator, SILENCE shows superior end-to-end inference speedup and privacy protection under different attack scenarios.Limitations: While for the first time, SILENCE provides a feasible privacy-preserving solution for resource-constrained audio devices, it introduces a huge design space for mask generator structures. The mask generator is akin to a lock; a genius lock design can protect privacy in the smallest of spaces, but a poor lock design can be bulky and easily broken. In this work, we simply inherit the SLU model structure and instantiate three sub-models from it to demonstrate better efficiency than previous encoders. Researchers can explore other structures for a better privacy-accuracy-efficiency trade-off. We will open-source all the code and checkpoints to facilitate further research in this direction.Some other potential limitations about lossy privacy-preserving capacity, the need for fine-tuning the cloud SLU model, the scope of defended threat model and the extension to offline scenarios are thoroughly discussed below for further clarification.Is current privacy-preserving capacity enough? The quantitative WER 80% is considered secure enough, as previous encoders have strived to reach that level [55, 11] . And some SLU transcripts contain the intent word, so the successfully inferred word might be a non-private intent word. For instance, in one test audio transcript, "I want some jazz music to play", the intent is 'scenario': 'play', 'action': 'music'. The interpretation of the malicious cloud ASR, "all subjects were used to play", is acceptable since the predicted phrase "to play" contains no private information. This scenario is typical for most audios; we managed to preserve 90% of the private entities in Figure 7 . This achievement matches the SoTA in privacy-preserving capacity, with up to 30× lower latency and 100× memory reduction.Why and how to fine-tune the cloud SLU Model? Initially, the cloud SLU is a generic pre-trained speech model lacking the capability to accurately understand personalized user intent. It is crucial to fine-tune the cloud SLU for better personalized intent understanding3 . Secondly, while shortdependent masking does not eliminate intent information, it does impact specific details within the attention map, as depicted in Figure 4 (b). Fine-tuning the cloud SLU model helps mitigate this impact and enhances the understanding of the user's intent. Currently, cloud service providers have already offered APIs that allow users to fine-tune their personalized cloud speech model [8] .Could private semantic detection attack be prevented? We clarified that detecting shortdependent key phrases or specific commands is not the focus of this work. For example, eavesdropping on specific financial words and political framing are out-of-scope. However, we can offer defense capabilities against them. The mask generator, controlled by the user, is trained to scrub utterances unrelated to the public intent. Private entities not predefined by the user are almost never included in the masked audio. Therefore, even if an attacker possesses a well-defined semantic and the mask generator, training the detection threat model is challenging because the synthetic masked audio lacks clear representations of the private semantic.Extesion to offline scenarios: Offline conditions occur periodically for resource-constrained devices. SILENCE can be easily integrated into an orchestration of small on-device SLU models and robust cloud models. This orchestration has been officially adopted by many off-the-shelf products, such as Apple Intelligence in iOS 18. Our system remains indispensable in such circumstances because small on-device SLU models may not generate satisfactory intent understanding due to their restricted model size. Even when on-device SLU models produce correct intent understanding, they cannot always operate due to limited device energy. As a result, online procedures are still the main components of current SLU solutions. The on-device functionality can be used as an alternative in offline conditions. With our system, the cloud-based SLU component is both privacy-preserving and efficient. Detailed analysis of FSC dataset. We conducted further experiments on the Fluent Speech Commands (FSC) dataset, another widely used dataset for spoken language understanding research. The FSC dataset includes 97 speakers and 30,043 relevant utterances. We split the data, using 20% for testing and the remaining 80% for training. The results are shown in Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).(11) Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper is intended for privacy protection and does not involve high-risk data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.(12) Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have properly cited the original code, data and models in §4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Note that these encoders are not specifically transformer encoders; rather, they can be implemented using any NNs to encode speech signals.The average duration of test SLU snippets is 2.8 seconds, with a maximum of 21.5 seconds, which is shorter than the maximum input window of speech models (e.g.,seconds for Whisper[45]).Note that a general speech model is sufficient for training the local mask generator in Figure5step (1a), as the focus is not on generating precise intent but rather on obtaining a coarse-grained distribution of numerical logits to facilitate mask generator training. 


Decision: The paper presents a  speech processing system that selectively obscures short-term details to reduce privacy leakage.  The proposed solution, a differential mask, is is based on the observation that speech understanding hinges on long-term dependency knowledge of the entire utterance, in contrast to privacy-sensitive elements that are short-term dependent. The authors propose a differential mask generator that automatically masks out portions of audio signals during online cloud inference. Empirical results show that the proposed system offers comparable speech understanding performance and privacy protection capacity with high memory efficiency.

The extensive additional experiments that that the authors have conducted during the rebuttal phase should be included in the paper. These include the illustrative description of different attack scenarios, visualizations of generated masks and reconstructed waveforms, and additional experiment results against advanced active reconstruction attacks, as well as the evaluation on the Fluent Speech Commands (FSC) dataset, the evaluation of the conventional modularised SLU and the different speech granularity.

The authors are also encouraged to replace "wimpy" with "resource-constrained", as it would be easier for the community to infer what they mean. Regardless, they should explain precisely and clearly what resources are deemed constrained, to what extent (i.e., in a quantifiable fashion), and why devices such as, e.g., Rasberry PIs pass/fail the definition test. Finally, active adversaries should also be explicitly addressed, both in the paper's presentation but also through the inclusion of the conducted additional experiment results against advanced active reconstruction attacks.

Reviews: review: paper_summary: paper_summary: This paper proposed a private speech processing system that selectively obscures short-term details to reduce privacy leakage in cloud-based Spoken Language Understanding (SLU). A differential mask generator is learned to automatically mask out portions of audio signals along with online cloud inference with the generated masks. Empirical results show that the proposed system offers comparable speech understanding performance and privacy protection capacity with high memory efficiency.
strength_weakness: strength_weakness: Could provide more background info for attack scenarios.
questions: questions: What types of devices can be defined as 'wimpy' devices?
limitations: limitations: N/A
paper_summary: paper_summary: This paper presents a lightweight speech intent understanding paradigm for wimpy devices, with heavy concern on the privacy. It targets the recently-popular disentanglement-based approaches for speech processing. It reached a decent balance between efficiency and privacy preservation, and make the SLU system work on wimpy devices.
strength_weakness: strength_weakness: Since this is more like a engineer-oriented work, the reviewer does not have strong problem or weakness from technical point of view.

But the reviewer does have a big problem on the architecture of SILENCE. Seems like from Figure 5, it still relies on cloud technology to maintain part of the framework. However, the wimpy devices sometimes (if not mostly) will meet offline conditions. In such case, the paper does not have an intention of making the system work.
questions: questions: 1. Compared to conventional SLU systems, do you think end-to-end systems preserve the nature of enhanced privacy? Do you think your algorithm will work for both the end-to-end system and the conventional modularized system?
2. The reviewer has problem with the definition of wimpy devices, since raspberry PI shall not be considered as "wimpy". Chips such as ARM cortex V7 has much lower memory and a lot of start-up are still using it for their offline solutions. The reviewer thinks "resource-constrained conditions" might be a better term. Of if there is official or your clear definition of "wimpy" devices with detailed stats, the reviewer is happy to keep head down and learn.
3. The reviewer are interested into the proposed framework with conventional methods.
limitations: limitations: The reviewer does not see any limitation or ethical concern about the paper.
paper_summary: paper_summary: The paper presents SILENCE, a method designed to address privacy concerns in cloud-based speech services by selectively obscuring short-term dependencies in speech signals. This technique preserves speech understanding functionality while protecting sensitive information. Implemented on the STM32H7 microcontroller, SILENCE significantly enhances speed and memory efficiency compared to existing solutions, effectively protecting privacy against various attack scenarios. Key contributions of the paper include an innovative encoder design based on asymmetric dependencies, the integration of automated masking configuration, and the demonstration of SILENCE's practical feasibility on low-resource devices.
strength_weakness: strength_weakness: - While the paper evaluates the system under black-box and white-box attacks, it does not thoroughly address the robustness of the masking approach against adaptive adversaries who might use advanced techniques to reverse-engineer the mask patterns.

- Masking short-term dependent frames to protect privacy could lead to losing crucial phonetic information necessary for accurate SLU. There is a lack of detailed analysis on how the granularity of the mask affects both privacy and SLU performance at different levels of speech granularity.

- The method relies heavily on the assumption that SLU tasks are primarily long-term dependent while ASR tasks are short-term dependent. This binary distinction might not hold for all SLU tasks, especially those involving contextually rich and intricate utterances.

- The experiments are primarily conducted on a single dataset. Evaluating the method on a broader range of datasets would better demonstrate its generalizability and robustness.
questions: questions: - How does SILENCE perform against adaptive adversaries using advanced reverse-engineering techniques?
- How does mask granularity affect both privacy protection and SLU performance at various levels of speech granularity?
- How does SILENCE handle SLU tasks that require short-term dependencies?
- How does SILENCE perform on additional datasets to ensure its generalizability and robustness across different speech scenarios?
limitations: limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper investigates the perturbation of audio signals to impair the accuracy of automatic speech recognition (ASR) models, thus enhancing privacy protection, while ensuring that spoken language understanding (SLU) models maintain high accuracy for interpreting user intentions. The key insight is that ASR models depend on local utterances, whereas SLU models utilize global features. The authors propose masking certain utterances to break local dependencies and explore the development of a learnable mask generator.
strength_weakness: strength_weakness: 1. My primary concern is the paper's limitation to passive adversaries. An active adversary might develop models specifically to reconstruct the raw signal from the masked signal. Such a threat is not investigated in this paper.
2. While the disruption of local dependencies does not affect the performance of user intent classification, this approach may not apply to all scenarios. Certain tasks, such as detecting key phrases or specific commands such as “set my clock at 10 AM” rely on local utterances.
questions: questions: 1. Could the authors elaborate on the potential ASR accuracy when facing an active adversary that trains a model to reconstruct raw signals from the masked signals?
2. Is it possible to visualize some masks generated by the trained mask generator to verify that they effectively disrupt local utterances, aligning with the main motivation of the study?
limitations: limitations: N/A

Comments: title: Correction to Reference in Response to Q1.2
comment: Dear reviewer,

We sincerely appreciate your thorough review of our rebuttal and the increased score. We kindly request further guidance on any remaining concerns you may have.

We are more than willing to provide additional clarification to enhance your understanding and satisfaction with our work.

Thank you once again for your recognition and assistance.

Best regards,

Authors
Thank you for the reply and for adding the new experiments. They address some of my concerns and I have increased my score to 5.
Dear Reviewer, 

We apologize for the typo in our previous communication regarding the reference for “Added implementation and experiments results of conventional modularized system.” The details are provided in **Q3** instead of **Q5**.

Best regards,

Authors
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your concerns and questions (see our rebuttal in the top-level comment and above), and revised our paper by following suggestions from all reviewers.

Additionally, we have included more references to underscore the importance of our focus on long-dependent intent classification in SLU literature [1-9].

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.

Best regards,

Authors

```
[1]. Rongxiang Wang and Felix Xiaozhu Lin. Turbocharge Deep Speech Understanding on the Edge. to appear at Proc. ACM Int. Conf. Mobile Computing and Networking (MobiCom), 2024.
[2]. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[3]. Jixuan Wang, Martin Radfar, Kai Wei and Clement  Chung. End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoder. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[4]. Bhuvan Agrawal, Markus Muller, Samridhi Choudhary, Martin Radfar, Athanasios Mouchtaris, Ross McGowan, Nathan Susanj, and Siegfried Kunzmann. Tie your embeddings down: Cross-modal latent spaces for end-to-end spoken language understanding. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[5]. Libo Qin, Tianbao Xie, Wanxiang Che and Ting Liu. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[6]. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A Spoken Language Understanding Resource Package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[7]. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253, 2017.
[8]. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[9]. Renato De Mori. Spoken language understanding: A survey. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 365–376. IEEE, 2007.
```
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your concerns and questions (see our rebuttal in the top-level comment and above), and revised our paper by following suggestions from all reviewers.

Additionally, we have included more references to underscore the importance of our focus on long-dependent intent classification in SLU literature [1-9].

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.

Best regards,

Authors

```
[1]. Rongxiang Wang and Felix Xiaozhu Lin. Turbocharge Deep Speech Understanding on the Edge. to appear at Proc. ACM Int. Conf. Mobile Computing and Networking (MobiCom), 2024.
[2]. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[3]. Jixuan Wang, Martin Radfar, Kai Wei and Clement  Chung. End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoder. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[4]. Bhuvan Agrawal, Markus Muller, Samridhi Choudhary, Martin Radfar, Athanasios Mouchtaris, Ross McGowan, Nathan Susanj, and Siegfried Kunzmann. Tie your embeddings down: Cross-modal latent spaces for end-to-end spoken language understanding. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[5]. Libo Qin, Tianbao Xie, Wanxiang Che and Ting Liu. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[6]. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A Spoken Language Understanding Resource Package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[7]. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253, 2017.
[8]. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[9]. Renato De Mori. Spoken language understanding: A survey. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 365–376. IEEE, 2007.
```
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your concerns and questions (see our rebuttal in the top-level comment and above), and revised our paper by following suggestions from all reviewers.

Additionally, we have included more references to underscore the importance of our focus on long-dependent intent classification in SLU literature [1-9].

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.

Best regards,

Authors

```
[1]. Rongxiang Wang and Felix Xiaozhu Lin. Turbocharge Deep Speech Understanding on the Edge. to appear at Proc. ACM Int. Conf. Mobile Computing and Networking (MobiCom), 2024.
[2]. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[3]. Jixuan Wang, Martin Radfar, Kai Wei and Clement  Chung. End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoder. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[4]. Bhuvan Agrawal, Markus Muller, Samridhi Choudhary, Martin Radfar, Athanasios Mouchtaris, Ross McGowan, Nathan Susanj, and Siegfried Kunzmann. Tie your embeddings down: Cross-modal latent spaces for end-to-end spoken language understanding. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[5]. Libo Qin, Tianbao Xie, Wanxiang Che and Ting Liu. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[6]. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A Spoken Language Understanding Resource Package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[7]. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253, 2017.
[8]. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[9]. Renato De Mori. Spoken language understanding: A survey. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 365–376. IEEE, 2007.
```
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your concerns and questions (see our rebuttal in the top-level comment and above), and revised our paper by following suggestions from all reviewers.

Additionally, we have included more references to underscore the importance of our focus on long-dependent intent classification in SLU literature [1-9].

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.

Best regards,

Authors

```
[1]. Rongxiang Wang and Felix Xiaozhu Lin. Turbocharge Deep Speech Understanding on the Edge. to appear at Proc. ACM Int. Conf. Mobile Computing and Networking (MobiCom), 2024.
[2]. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[3]. Jixuan Wang, Martin Radfar, Kai Wei and Clement  Chung. End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoder. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[4]. Bhuvan Agrawal, Markus Muller, Samridhi Choudhary, Martin Radfar, Athanasios Mouchtaris, Ross McGowan, Nathan Susanj, and Siegfried Kunzmann. Tie your embeddings down: Cross-modal latent spaces for end-to-end spoken language understanding. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[5]. Libo Qin, Tianbao Xie, Wanxiang Che and Ting Liu. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[6]. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A Spoken Language Understanding Resource Package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[7]. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253, 2017.
[8]. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[9]. Renato De Mori. Spoken language understanding: A survey. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 365–376. IEEE, 2007.
```